Senior Data Engineer (Databricks)

DataGraders is a consultancy specializing in data warehousing, business intelligence, and business planning services that enable organizations to transform data into actionable insights. The team consists of seasoned professionals with expertise across a broad spectrum of analytics and BI solutions. They tailor strategies and solutions to each client’s unique business objectives, guiding projects from requirements gathering through to the implementation of robust and scalable architectures. DataGraders is dedicated to helping businesses fully leverage their data assets, driving measurable improvements in performance and decision-making.

The Senior Data Engineer (Databricks) role is a full-time, on-site position based in Lahore. The successful candidate will design, build, and optimize scalable data pipelines and platforms to support advanced analytics and reporting needs. This role requires close collaboration with data analysts, data scientists, and business stakeholders to understand their requirements and translate them into efficient data models and ETL workflows. The engineer will develop and maintain Databricks-based solutions, integrate data from multiple sources, ensure data quality and reliability, and implement best practices in data warehousing and governance. Additionally, the role involves monitoring system performance, troubleshooting issues, and contributing to the continuous improvement of DataGraders’ data engineering standards and frameworks.

Key Responsibilities
- Design, build, and optimize scalable data pipelines and platforms using Databricks and related technologies.
- Collaborate with data analysts, data scientists, and business stakeholders to gather requirements and translate them into efficient data models and ETL processes.
- Develop and maintain ETL workflows, ensuring data integration from multiple sources is seamless and reliable.
- Ensure data quality, consistency, and reliability across data pipelines and storage solutions.
- Implement best practices in data warehousing, including schema design, data governance, and security protocols.
- Monitor system performance, identify bottlenecks, and troubleshoot data pipeline issues promptly.
- Contribute to the enhancement of data engineering standards, frameworks, and documentation within the organization.

Required Qualifications
- Proven experience in building, maintaining, and optimizing large-scale data pipelines and distributed data processing systems.
- Strong skills in data modeling and designing scalable, maintainable schemas suitable for analytics and reporting environments.
- Hands-on expertise with ETL processes, including orchestration, automation, and performance tuning.
- Solid understanding of data warehousing concepts such as star and snowflake schemas, and experience with modern cloud data platforms.
- Applied data analytics capabilities to support BI teams by delivering clean, reliable, and well-structured data.
- Advanced proficiency with Databricks and Apache Spark, along with experience on at least one major cloud platform (AWS, Azure, or GCP).
- Strong command of SQL and programming languages commonly used in data engineering, such as Python or Scala.
- Knowledge of data governance, security best practices, and techniques for data quality monitoring and documentation.
- Ability to work independently with minimal supervision while collaborating effectively across cross-functional teams.
- Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Engineering, or a related field, or equivalent practical experience.

This role offers the opportunity to join a forward-thinking consultancy committed to client success in a dynamic, data-driven environment. Candidates will be part of a skilled team focused on delivering tailored data solutions that empower businesses to make informed decisions and achieve measurable results.