Data Quality Engineer

We are seeking a skilled Data Quality Engineer with strong expertise in Data Quality frameworks and proven experience in implementation for a 60-working-day enterprise Data Quality and Governance program at a government entity in Abu Dhabi. This full-time role involves hands-on design, implementation, and validation of Data Quality controls across extensive government datasets, ensuring adherence to data governance standards and DAMA principles. The engineer will work with tools such as Microsoft Purview, Azure Databricks, PySpark, SQL, and Informatica IDQ to profile data, author rules, automate measurements, analyze root causes, and drive remediation efforts. The position requires immediate availability or within 1–2 weeks and is based in Abu Dhabi with a hybrid working pattern as directed.

Key Responsibilities

- Profile source datasets and collaborate with business and data owners to identify Critical Data Elements (CDEs).
- Design, author, and validate Data Quality rules covering Completeness, Validity, Consistency, Uniqueness, Accuracy, and Timeliness.
- Implement Data Quality rules at scale using Microsoft Purview, executing validations on Azure Databricks with PySpark and SQL; utilize Informatica IDQ when necessary.
- Develop Python automation scripts leveraging Purview and Databricks APIs for profiling, reconciliation, bulk rule authoring, and metadata remediation.
- Build and maintain an automated Data Quality measurement and reporting framework that extracts rule definitions, executes validations, and calculates table-level Data Quality Index (DQI) scores.
- Conduct root-cause analysis on failed records, review rule thresholds, investigate source data issues, and manage rule-to-column mappings to support remediation.
- Prepare reusable playbooks, Data Quality rule standards, and data classification guidelines aligned with client and UAE data governance policies.
- Support data cataloging activities including business glossary, data lineage, ownership, and classification alongside Data Quality delivery.
- Provide weekly progress reports, issue updates, and quality results to the engagement lead, and submit timesheets accordingly.

Required Qualifications

- Comprehensive understanding of Data Quality frameworks, including profiling, CDE identification, rule design, measurement, scoring, exception management, and remediation.
- Minimum 4–6 years of data experience, with at least 2 years of hands-on Data Quality rule design, implementation, and validation on enterprise or government programs.
- Proficiency in Microsoft Purview, including Data Quality rules, rule-to-column bindings, scans, data catalog, business glossary, and classification.
- Strong experience with Azure Databricks, PySpark, Spark SQL, Delta Lake, ADLS Gen2, and Unity Catalog.
- Skilled in Python programming and REST API automation for profiling, reconciliation, and bulk rule operations.
- Advanced SQL querying capabilities for validation, reconciliation, and root-cause analysis.
- Working knowledge of Informatica IDQ; familiarity with Enterprise Data Catalog (EDC) or Axon is advantageous.
- Deep understanding of Data Quality dimensions: Completeness, Validity, Consistency, Uniqueness, Accuracy, and Timeliness, including threshold setting and tuning.
- Knowledge of DAMA-DMBOK principles, metadata management, data lineage, ownership, and classification.
- Excellent communication skills in English, with the ability to collaborate effectively with client stakeholders through the engagement lead.
- Compliance with government client requirements such as background checks, security clearance or screening, and confidentiality agreements.
- Valid UAE driving license and willingness to travel to client sites as required.

Preferred Qualifications and Benefits

- Experience with UAE or Abu Dhabi government data governance standards or work aligned with the Data Governance Emirates (DGE) framework.
- Certifications such as Databricks Certified Data Engineer Associate or Professional.
- Azure certifications including AZ-900 or higher-level credentials like DP-203 or DP-700.
- Familiarity with Azure Data Factory, Azure DevOps, Git, and CI/CD pipelines.
- Arabic language skills are a plus.
- Domain experience in banking, pharmaceuticals, or the public sector.

Deliverables

- Profiling results and a comprehensive CDE register for the scoped datasets.
- Validated Data Quality rules with accurate rule-to-column bindings.
- Automated Data Quality measurement and reporting system delivering table-level DQI scores.
- Root-cause analysis documentation and remediation notes for failed rules.
- Reusable playbooks, rule standards, and classification guidelines aligned with client and UAE standards.

This opportunity offers a challenging and impactful role within a high-profile government data governance initiative, ideal for candidates with a strong technical background and a commitment to data quality excellence.