Principal Consultant - Data Analytics

Job Summary
We are seeking a skilled Data Engineer with extensive experience in designing, developing, and maintaining scalable data pipelines using Azure Data Factory. The ideal candidate will play a pivotal role in building robust data solutions by leveraging Azure DevOps for CI/CD implementations and utilizing Microsoft Fabric to enhance data warehousing and analytics capabilities. This role requires a strong background in ETL development with PySpark, ensuring data integrity and availability, and collaborating effectively with cross-functional teams to drive data initiatives forward.

Key Responsibilities
Design, develop, and maintain scalable and efficient data pipelines using Azure Data Factory to support various business needs. Implement continuous integration and continuous deployment (CI/CD) pipelines through Azure DevOps, ensuring seamless delivery and deployment of data solutions. Utilize Microsoft Fabric to improve data warehousing infrastructure and enhance analytics performance. Develop, optimize, and maintain ETL processes primarily using PySpark, focusing on distributed data processing for large datasets. Ensure high standards of data integrity, quality, and availability across multiple systems and platforms. Collaborate closely with data scientists, analysts, and product managers to understand data requirements and deliver actionable insights. Monitor, troubleshoot, and resolve issues within data workflows to maintain optimal performance and reliability. Maintain comprehensive documentation and adhere to best practices in data engineering to support ongoing maintenance and scalability.

Required Qualifications
Demonstrated expertise in Azure Data Factory for building and managing data pipelines. Proven experience with Azure DevOps, particularly in setting up and managing CI/CD pipelines for data-related projects. Hands-on experience working with Microsoft Fabric to support data warehousing and analytics solutions. Proficiency in PySpark and distributed data processing frameworks, enabling efficient ETL development and optimization. Strong understanding of data warehousing concepts, cloud architecture, and best practices in data engineering. Excellent problem-solving skills with the ability to troubleshoot complex data workflows. Effective communication skills to collaborate with diverse teams and stakeholders. Ability to work independently while also contributing effectively within a team environment.

Preferred Qualifications and Benefits
While not explicitly stated, candidates with additional certifications in Azure or related cloud technologies, experience with other big data tools, or a background in data science collaboration will be highly valued. This role offers the opportunity to work with cutting-edge Microsoft technologies in a dynamic and collaborative environment, contributing to impactful data-driven business decisions.

If you are passionate about data engineering and eager to work with advanced cloud-based data platforms, we encourage you to apply and join our innovative team.