Principal Data Engineer - II

We are seeking an experienced Data Engineer to design, develop, and maintain scalable ETL/ELT pipelines that efficiently ingest, transform, and integrate data from multiple sources. The ideal candidate will implement both real-time data streaming and batch processing solutions to meet diverse business requirements. This role involves building and optimizing data storage solutions such as data lakes, warehouses, and databases while ensuring that all data systems are scalable, secure, and high-performing.

The Data Engineer will be responsible for cleaning, transforming, and preparing raw data into usable formats suitable for analytics and machine learning. Collaboration with Data Scientists, Analysts, and other cross-functional teams is essential to define data requirements and deliver effective solutions. Monitoring the performance and reliability of data pipelines, troubleshooting issues, and implementing improvements to minimize downtime are key aspects of this position.

Additionally, the role requires implementing data quality checks, validation processes, and monitoring mechanisms to maintain data integrity. Working closely with Data Governance teams to ensure compliance with organizational standards and policies is also critical. The successful candidate will provide technical expertise and guidance on data engineering best practices, stay informed about emerging tools and technologies, and evaluate new solutions to enhance data engineering capabilities.

Key Responsibilities

- Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and integration.
- Implement real-time data streaming and batch processing solutions tailored to business needs.
- Build and optimize data storage systems, including data lakes, warehouses, and databases.
- Ensure data systems and pipelines are scalable, secure, and performant.
- Clean, transform, and prepare raw data for analytics and machine learning applications.
- Collaborate with Data Scientists, Analysts, Data Architects, and Business Teams to understand requirements and deliver data solutions.
- Monitor pipeline and system performance, troubleshoot issues, and implement necessary improvements.
- Implement and enforce data quality checks, validation processes, and monitoring mechanisms.
- Work with Data Governance teams to ensure compliance with data standards and policies.
- Provide technical guidance on data engineering best practices.
- Stay updated on emerging tools, technologies, and industry trends.
- Evaluate and recommend technologies to enhance data engineering capabilities.

Required Qualifications

- Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
- 7 to 8 years of experience as a Data Engineer or in a similar role, with proven expertise in building and maintaining data pipelines and infrastructure.
- Hands-on experience with big data tools, cloud platforms, and data integration techniques.
- Proficiency in SQL and experience working with both relational and NoSQL databases.
- Experience with data orchestration tools such as ODI, SSIS, Apache Airflow, dbt, or Prefect.
- Strong programming skills in Python, Java, or Scala.
- Familiarity with data processing frameworks like Apache Spark and Kafka.
- Excellent problem-solving and analytical abilities.
- Strong communication and collaboration skills.
- Ability to thrive in a fast-paced, team-oriented environment.

Preferred Qualifications and Benefits

Candidates with additional experience in emerging data technologies and a proactive approach to continuous learning will be highly valued.

We offer a competitive salary package along with a fuel card and comprehensive health benefits. Professional development opportunities are provided to support career growth. Our inclusive work culture fosters collaboration and innovation, making this an excellent environment for motivated data professionals.