Lead II - Data Engineering

UST is a global digital transformation solutions company dedicated to helping leading organizations grow and succeed. With a workforce of over 30,000 employees across more than 30 countries, UST combines talent, technology, and innovation to co-create lasting change that impacts billions of lives worldwide. We are currently seeking an experienced Big Data Developer to join our team and contribute to building scalable, high-performance data solutions.

Key Responsibilities

- Design, build, and operate production data pipelines primarily on AWS, ensuring reliability and scalability.
- Develop production-quality code using Python or Scala, adhering to best practices and coding standards.
- Write and optimize complex SQL queries, including advanced techniques such as window functions, Common Table Expressions (CTEs), and query plan analysis.
- Utilize deep hands-on experience with Apache Spark for data processing, including performance tuning, debugging, and resolving issues like data skew, shuffle bottlenecks, and memory constraints.
- Manage and operate AWS data stack components such as EMR, S3, Athena, Redshift, and Lambda in production environments.
- Handle streaming data ingestion and processing using Amazon Kinesis or Apache Kafka.
- Implement workflow orchestration using Apache Airflow (MWAA) or similar scheduling tools.
- Apply solid data modeling principles, including dimensional modeling and handling slowly changing dimensions.
- Work with modern data lake table formats such as Apache Iceberg, Hudi, or Delta Lake.
- Monitor and maintain data pipelines using tools like CloudWatch and structured logging, including participation in on-call rotations.
- Collaborate effectively with non-technical stakeholders, clearly communicating technical trade-offs and solutions.
- Follow version control best practices and participate in collaborative development workflows using Git.

Required Qualifications

- 7 to 10 years of professional experience in data engineering or related fields.
- At least 4 years of hands-on experience building and managing production data pipelines on AWS.
- Strong programming skills in Python or Scala with a focus on writing maintainable, production-ready code.
- Expert-level proficiency in SQL, including advanced query techniques and performance tuning.
- Extensive experience with Apache Spark, including troubleshooting and optimizing complex data workflows.
- Proven ability to diagnose and resolve Spark-specific performance issues such as data skew and memory management.
- Practical experience with AWS data services including EMR, S3, Athena, Redshift, and Lambda.
- Experience in streaming data technologies like Amazon Kinesis or Apache Kafka.
- Familiarity with workflow orchestration tools such as Apache Airflow or equivalents.
- Solid understanding of data modeling concepts and experience with data lake table formats.
- Experience in operating data pipelines with monitoring, logging, and incident response.
- Proficient with Git and collaborative development environments.
- Strong communication skills to engage with both technical and non-technical stakeholders.

Preferred Qualifications and Benefits

While not explicitly listed, candidates with additional expertise in cloud-native data architectures, advanced data governance, or experience with emerging data lake technologies will find opportunities for growth. UST fosters a culture rooted in humility, humanity, and integrity, encouraging innovation, diversity, and inclusion. Employees are supported to thrive in a collaborative environment that values ethical business practices and social responsibility.

UST is an Equal Opportunity Employer committed to creating an inclusive workplace free from discrimination based on age, disability, ethnicity, gender identity, religion, sexual orientation, or any other protected status. Employment decisions are made solely on merit, qualifications, and business needs.

Candidates joining UST will be expected to uphold company values, comply with policies, and remain adaptable to evolving roles and responsibilities aligned with organizational goals.

Technologies involved include AWS Lambda, Amazon Athena, Amazon CloudWatch, Amazon Kinesis, Amazon Redshift, Amazon S3, Apache Airflow, Apache Hudi, and Amazon EMR.