AWS Data Engineer

UST is seeking a Lead Data Engineer with expertise in AWS to join their dynamic team. This role involves developing scalable, API-driven systems to manage and monitor large-scale batch Big Data applications. The ideal candidate will design and maintain data transformation processes using technologies such as Spark, SQL, Hive, Python, and Scala, while leveraging AWS cloud services including S3, EC2, EMR, Lambda, DynamoDB, and API Gateway. Collaboration with cross-functional teams in an Agile environment is essential to deliver high-quality data engineering solutions that support operational workflows and improve system reliability.

Key Responsibilities:
- Develop API-driven systems to govern, manage, and monitor large-scale batch Big Data applications.
- Build scalable backend services and data engineering solutions to support data processing and operational workflows.
- Design, develop, and maintain data transformation pipelines using Spark, SQL, Hive, Python, and Scala.
- Utilize AWS cloud services such as S3, EC2, EMR, Lambda, DynamoDB, and API Gateway to create cloud-native data and backend solutions.
- Enhance workflow orchestration with Apache Airflow, including advanced DAG design, dependency management, scheduling, monitoring, and failure handling.
- Define technical scope, objectives, and implementation strategies by participating in requirements gathering, technical research, and process definition.
- Engage in architecture and code reviews, performance tuning, and operational readiness activities.
- Contribute to test planning and validation for application integrations and project deliverables.
- Collaborate with product owners, data engineers, backend engineers, QA, DevOps, and other teams to deliver solutions in an Agile/Scrum environment.
- Troubleshoot production issues, improve system reliability, and drive automation across data and backend workflows.

Required Qualifications:
- Minimum of 5 years of hands-on experience developing enterprise-scale applications, data platforms, or distributed systems.
- At least 5 years of experience working with Big Data platforms and cloud infrastructure, preferably AWS EMR and the Hadoop ecosystem.
- Proficiency in AWS services including S3, EC2, EMR, Lambda, DynamoDB, and API Gateway.
- Strong hands-on experience with Spark, Python and/or Scala, Apache Airflow, SQL, and Hive.
- Ability to build production-grade data pipelines and backend services using Python or Scala.
- Deep understanding of data engineering concepts such as data pipelines, batch and real-time processing, data quality, and performance optimization.
- Experience with version control and CI/CD tools like Git and Jenkins, supporting build automation, testing, and deployment workflows.
- Excellent analytical, problem-solving, communication, and presentation skills, capable of explaining technical concepts to diverse audiences.
- Proven ability to work effectively in cross-functional Agile/Scrum teams.

Preferred Qualifications:
- Experience with Large Language Models (LLMs), Generative AI, Agentic AI, or AI-assisted engineering workflows.
- Familiarity with API design, microservices, event-driven architecture, or serverless backend development.
- Knowledge of CI/CD pipelines, infrastructure as code, automated testing, and production deployment best practices.
- Experience in data governance, metadata management, lineage, data observability, or operational controls for enterprise data platforms.

Education:
- Bachelor’s degree in Computer Science, Engineering, Mathematics, Information Systems, or a related technical field, or equivalent practical experience.

UST is committed to fostering a culture of integrity, humility, and humanity. The company values diversity and inclusion, encouraging innovation and placing people at the center of everything it does. UST is an Equal Opportunity Employer, ensuring all employment decisions are made without regard to age, race, gender, disability, or other protected characteristics. Employees are expected to support and promote UST’s values and comply with company policies and procedures.

This role offers the opportunity to work with cutting-edge technologies in a collaborative environment, contributing to impactful data engineering projects that serve global clients.