Senior Data Engineer

This role involves building and operating the Databricks Lakehouse platform with a strong focus on data engineering, pipeline development, data modeling, and ensuring data quality. The candidate will implement the architecture and security frameworks as defined by the Data Architect, taking full ownership of the platform’s performance, reliability, and governance.

Key Responsibilities

- Build and maintain ETL/ELT pipelines that ingest and transform data from multiple sources, ensuring data readiness in terms of completeness, quality, granularity, latency, and access. Develop remediation plans where necessary.
- Design and develop Delta Lake pipelines across medallion layers (bronze, silver, gold) for both batch and streaming data processing.
- Provision and manage cloud data infrastructure using Terraform, leveraging automated, version-controlled Infrastructure as Code pipelines.
- Build and optimize data models tailored for reporting and analytics needs, while troubleshooting and optimizing SQL queries and data pipeline issues end-to-end.
- Own data validation, quality assurance, and lineage tracking within assigned data domains.
- Configure and manage Databricks workspaces, clusters, SQL warehouses, and jobs to ensure smooth platform operations.
- Implement Unity Catalog structures including catalogs, schemas, access policies, and lineage in alignment with the governance model.
- Configure row-level and column-level security, as well as data masking, following the Architect’s design.
- Monitor platform performance, cost efficiency, and reliability, making adjustments as needed.
- Maintain comprehensive runbooks and operational documentation to support ongoing platform management.
- Develop frameworks, abstraction layers, and tooling to empower data engineers and analysts to work efficiently on the platform.

Required Qualifications

- Minimum of 8 years of experience in data or platform engineering, including at least 3 years of hands-on production experience with Databricks.
- Strong proficiency in SQL and Python, particularly PySpark.
- Understanding of Genie spaces and their application.
- Solid working knowledge of data governance, metadata management, data cataloging, and data lineage concepts.
- Experience with Delta Lake, Unity Catalog, Databricks SQL, and managing workflows/jobs within Databricks.
- Hands-on experience with cloud data warehouses such as Snowflake and Redshift.
- Expertise in dimensional/star schema modeling for data warehousing.
- Proficiency with Terraform and CI/CD pipelines, especially using GitHub Actions.
- Experience using AI-assisted developer tools, including large language models (LLMs), Model Context Protocol (MCP), or copilot-style productivity tools.

Preferred Qualifications

- Experience working with AWS data and cloud technologies.
- Databricks certification such as Data Engineer Associate or Professional is highly desirable.
- Exposure to Generative AI (GenAI) or AI-driven Business Intelligence (AI-BI) use cases.
- Familiarity with distributed data processing frameworks, particularly Apache Spark.

This position offers the opportunity to work at the forefront of modern data engineering practices, leveraging cutting-edge tools and frameworks to build scalable, secure, and efficient data platforms. The ideal candidate will be proactive, detail-oriented, and passionate about enabling data-driven decision-making across the organization.