Job Description: Data Engineer II

Smart IMS is committed to creating a workplace that enriches the lives of its associates through meaningful experiences and a diverse, inclusive environment. We make hiring decisions free from bias or discrimination based on any protected characteristic. Our focus is on supporting your career growth and helping you succeed.

We are seeking a skilled Data Engineer II to join our Data Platform team. In this role, you will be instrumental in designing, developing, and optimizing next-generation data infrastructure. You will work with large-scale distributed systems, combining traditional big data engineering with modern cloud-native and AI-driven workflows to deliver high-throughput, low-latency data solutions.

Key Responsibilities

Data Pipeline Development & Optimization
Design, develop, and maintain scalable ETL/ELT pipelines using Scala and Apache Spark/Flink (Core, SQL, Streaming) to process massive datasets efficiently. Optimize Spark jobs and SQL queries to improve performance and resource utilization. Implement and manage data tables using modern Lakehouse formats such as Apache Iceberg, Hudi, or Delta Lake to ensure efficient data storage and retrieval.

Data Management & Quality
Apply Medallion Architecture principles (Bronze/Silver/Gold) to structure data for analytics and machine learning use cases. Implement data validation checks and automated testing frameworks like Deequ or Great Expectations to maintain data accuracy and reliability. Integrate observability tools to monitor data health, freshness, and lineage.

Cloud Native Engineering
Deploy and manage workloads on Google Cloud Platform’s DataProc and Kubernetes, leveraging containerization for scalable data processing. Contribute to infrastructure automation through scripting and infrastructure-as-code practices.

Collaboration & Innovation
Explore and implement Generative AI and agentic workflows to automate data discovery and enhance engineering processes. Work closely with architects and product teams in an Agile/Scrum environment to deliver iterative features. Participate in code reviews to uphold code quality and best practices.

Required Qualifications

Experience: 3-5 years of hands-on experience in data engineering roles.

Technical Skills:
- Strong proficiency in Scala and Apache Spark (batch and streaming).
- Solid understanding of SQL and distributed computing concepts.
- Experience with cloud platforms such as GCP (DataProc, GCS, BigQuery) or equivalents like AWS/Azure.
- Hands-on experience with Kubernetes and Docker.

Architecture & Storage:
- Experience working with Lakehouse table formats (Iceberg, Hudi, or Delta Lake).
- Understanding of data warehousing and modeling concepts, including Star and Snowflake schemas.

Soft Skills:
- Strong problem-solving abilities and the capacity to work independently.
- Effective communication skills for collaboration with cross-functional teams.

Education Qualification
Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related quantitative discipline.

Preferred Qualifications and Benefits

Preferred Qualifications:
- Familiarity with machine learning concepts, feature engineering, or building data pipelines for ML models.
- Experience with workflow orchestration tools such as Airflow or Azkaban.
- Knowledge of real-time analytics databases like Druid, ClickHouse, or HBase.
- Experience implementing CI/CD pipelines for data applications.

Why Join Us?
At Smart IMS, you will tackle petabyte-scale data challenges that set industry standards. Collaborate with top-tier engineers in a dynamic, high-growth environment. Gain exposure to cutting-edge technologies including Apache Iceberg, Kubernetes, and Generative AI.

Experience: 3-5 years
Location: Hyderabad (or as applicable)
Team: Data Platform & Engineering