Data Engineer (Azure)

We are seeking a skilled Data Engineer to design, build, and maintain data pipelines on the Azure platform. In this role, you will be responsible for managing the entire ETL process—from ingesting data from various sources to transforming it using SQL and code, and finally delivering reliable data to stakeholders and applications. You will collaborate closely with data scientists to ensure data accuracy, accessibility, and trustworthiness, while also automating quality checks to maintain data integrity. You will have the flexibility to select the most appropriate tools for each task, whether that involves Azure Data Factory or other technologies.

Key Responsibilities
- Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory, Synapse pipelines, Databricks, Azure Functions, or custom code as needed.
- Implement data ingestion workflows from diverse sources including databases, APIs, flat files, event streams, and third-party SaaS platforms, supporting both batch and near-real-time processing.
- Create and manage data endpoints such as APIs, views, and shared datasets to provide reliable data access for analysts, data scientists, and applications.
- Transform and model data with SQL, Python/PySpark, and other methods to produce clean, well-documented datasets ready for analysis.
- Collaborate with data scientists to automate data quality checks, validation rules, and alerts to proactively identify and resolve issues before they impact models or reports.
- Manage data access and permissions, ensuring appropriate and secure data availability.
- Monitor, troubleshoot, and optimize pipelines for reliability, performance, and cost-efficiency.
- Apply CI/CD practices and version control to data pipelines and infrastructure.
- Document data pipelines, sources, and definitions to facilitate knowledge sharing and future development.

Required Qualifications
- Minimum of 3 years experience as a Data Engineer or in a similar role focused on building production data pipelines.
- Hands-on experience with the Azure data ecosystem, including Azure Data Factory and at least one of Azure Synapse, Azure Databricks, or Microsoft Fabric.
- Strong proficiency in SQL, including complex querying, performance tuning, and data modeling techniques such as star schemas and slowly changing dimensions.
- Skilled in Python or a similar programming language for data processing and automation tasks.
- Experience ingesting data from multiple sources such as REST APIs, relational databases, and file-based feeds.
- Familiarity with building or exposing data through endpoints like REST APIs, Azure Functions, or curated views.
- Working knowledge of Azure storage and database services including ADLS Gen2, Azure SQL Database, and Blob Storage.
- Comfortable using Git and CI/CD tools such as Azure DevOps or GitHub Actions.
- Strong communication skills with the ability to work effectively with both technical and non-technical stakeholders.

Preferred Qualifications and Benefits
- Experience with data sanitation processes including cleansing, deduplication, standardization, and anonymization of sensitive data (PII).
- Knowledge of data governance tools such as Microsoft Purview for cataloging, lineage, and classification.
- Understanding of data privacy and compliance frameworks like GDPR, HIPAA, or SOC 2.
- Familiarity with data quality frameworks such as Great Expectations, dbt tests, or Soda.
- Experience with Azure security and access controls including Entra ID (Azure AD), RBAC, Key Vault, managed identities, and row-level security.
- Exposure to streaming technologies such as Azure Event Hubs, Stream Analytics, or Kafka.
- Experience with infrastructure as code tools like Terraform, Bicep, or ARM templates.
- Support experience for machine learning workflows, including feature pipelines or Azure Machine Learning.
- Relevant Azure certifications such as Microsoft Certified: Fabric Data Engineer Associate or Azure Data Engineer Associate are a plus.

Tools You Will Use
- Orchestration & ETL: Azure Data Factory, Synapse pipelines, Databricks, Azure Functions
- Storage & Databases: ADLS Gen2, Azure SQL Database, Synapse, Blob Storage
- Languages: SQL, Python, PySpark
- Endpoints & Integration: REST APIs, Azure Functions, API Management
- Quality & Governance: Microsoft Purview, data quality frameworks
- Security & Access: Entra ID, RBAC, Key Vault
- DevOps: Git, Azure DevOps, GitHub Actions

This position requires working on-site to collaborate effectively with the team and stakeholders. If you are passionate about building scalable, reliable data infrastructure and enjoy working in a dynamic environment, we encourage you to apply.