Data Engineer II

Smart IMS is looking for a talented Data Engineer II to join our Data Platform team. In this role, you will play a key part in designing and enhancing next-generation data infrastructure that supports high-throughput distributed systems. You will work with large-scale data environments, combining traditional big data engineering techniques with modern cloud-native and AI-driven workflows. We pride ourselves on fostering an inclusive and diverse workplace that encourages career growth and success through collaboration.

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines using Scala and Apache Spark or Flink (Core, SQL, Streaming) to handle large datasets with minimal latency. Continuously optimize Spark jobs and SQL queries to enhance performance, reduce resource consumption, and increase processing speed. Manage data tables using modern Lakehouse formats such as Apache Iceberg, Hudi, or Delta Lake to ensure efficient storage and retrieval.

Apply Medallion Architecture principles (Bronze, Silver, Gold) to organize data for downstream analytics and machine learning applications. Implement data validation and automated testing frameworks like Deequ or Great Expectations to maintain data accuracy and reliability. Integrate pipelines with observability tools to monitor data health, freshness, and lineage effectively.

Deploy and manage workloads on Google Cloud Platform (GCP) DataProc and Kubernetes (K8s), leveraging containerization for scalable data processing. Contribute to infrastructure automation and develop deployment scripts to support infrastructure as code best practices.

Explore and implement Generative AI (GenAI) and Agentic workflows to automate data discovery and improve engineering efficiency. Collaborate closely with architects and product teams in an Agile/Scrum environment to deliver iterative features. Participate in code reviews to uphold high standards of code quality and best practices.

Required Qualifications

3 to 5 years of hands-on experience in Data Engineering roles. Strong proficiency in Scala and Apache Spark for both batch and streaming data processing. Solid understanding of SQL and distributed computing concepts. Experience with GCP services such as DataProc, Google Cloud Storage (GCS), and BigQuery, or equivalent cloud platforms like AWS or Azure. Practical experience with Kubernetes and Docker containerization technologies. Familiarity with Lakehouse table formats including Iceberg, Hudi, or Delta Lake. Knowledge of data warehousing and modeling concepts, including Star and Snowflake schemas. Strong problem-solving skills with the ability to work independently. Effective communication skills to collaborate with cross-functional teams. Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related quantitative discipline.

Preferred Qualifications and Benefits

Experience in machine learning, including familiarity with ML concepts, feature engineering, or building data pipelines for ML models. Knowledge of workflow orchestration tools such as Airflow or Azkaban. Exposure to real-time analytics databases like Druid, ClickHouse, or HBase. Experience implementing CI/CD pipelines for data applications. Opportunity to work on petabyte-scale data challenges that influence industry standards. Collaborate with top-tier engineers in a dynamic, fast-growing environment. Gain exposure to cutting-edge technologies including Apache Iceberg, Kubernetes, and Generative AI.

Location: Hyderabad (or as applicable)
Team: Data Platform & Engineering

Join Smart IMS to advance your career while contributing to innovative data engineering solutions in a supportive and inclusive workplace.