Lead Data Engineer (AWS Data Platform)

Position: Lead Data Engineer (AWS Data Platform)
Experience: 5 to 8+ Years
Location: Hybrid (Pakistan)
*Note: Candidates must be available during UAE business hours and observe UAE public holidays.*

We are looking for a highly experienced Lead Data Engineer with over 5 years of expertise in designing, developing, and optimizing large-scale data platforms and ETL/ELT pipelines. The ideal candidate will have strong hands-on skills in PySpark, AWS Glue, Amazon EMR, Amazon Redshift, and SQL-based data warehousing. Proven experience in performance tuning and data optimization is essential. This role involves close collaboration with a UAE-based client, requiring the candidate to work within UAE business hours and coordinate effectively with distributed teams.

Key Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark, AWS Glue, and Amazon EMR.
- Build and optimize data ingestion, transformation, and processing frameworks for both structured and semi-structured data.
- Develop and manage enterprise data warehouse solutions leveraging Amazon Redshift.
- Write complex SQL queries, stored procedures, and data transformations to support analytics and reporting needs.
- Implement efficient ETL/ELT processes to move data across multiple systems and platforms.
- Perform performance tuning and optimization of Spark jobs, ETL pipelines, SQL queries, and Redshift workloads.
- Ensure data quality, integrity, security, and governance across all data platforms.
- Troubleshoot production issues and conduct root cause analysis for data-related incidents.
- Collaborate with business stakeholders, analysts, architects, and engineering teams to gather and understand data requirements.
- Participate in code reviews, technical design discussions, and enforce best practices.
- Monitor data pipelines proactively to identify opportunities for performance improvements and automation.
- Create and maintain technical documentation, data models, and operational procedures.

Required Qualifications
- 5 to 8+ years of experience in Data Engineering and Data Warehousing.
- Strong hands-on expertise with PySpark and AWS Glue.
- Experience building and managing workloads on Amazon EMR.
- Deep knowledge of Amazon Redshift and SQL development including query optimization.
- Solid understanding of Data Warehousing concepts, dimensional modeling, and ETL/ELT processes.
- Proven ability in performance tuning of Spark jobs, SQL queries, ETL pipelines, and data warehouse workloads.
- Experience handling large-scale datasets and distributed data processing environments.
- Excellent debugging, troubleshooting, and analytical skills.
- Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or a related field.
- Relevant AWS certifications are a plus.

Preferred Qualifications and Additional Skills
- Familiarity with other AWS services such as S3, IAM, CloudWatch, Lambda, and Step Functions.
- Knowledge of CI/CD pipelines and DevOps practices tailored for data platforms.
- Experience with workflow orchestration tools.
- Understanding of data governance, security, and compliance frameworks.
- Exposure to Agile/Scrum development methodologies.
- Strong communication and stakeholder management skills.
- Ability to work independently in a remote or hybrid environment.
- Excellent problem-solving and analytical thinking capabilities.
- Proven ability to collaborate effectively with cross-functional and geographically distributed teams.
- Demonstrates strong ownership mindset and commitment to delivering high-quality solutions.

This role offers the opportunity to lead critical data engineering initiatives within a dynamic environment, working closely with international teams and leveraging cutting-edge AWS technologies. Candidates must be adaptable to UAE working hours and public holidays to ensure seamless collaboration with the client.