Senior Data Engineer

The Data Enablement team plays a critical role in supporting the entire organization with its data needs. From the moment data enters the platform until it reaches the intended users, this small, senior team of data and AI engineers manages customer data, product knowledge, and operational data. The team ensures that data is unified into a reliable single source, accessible only to authorized users, and verifiably accurate. As a Senior Data Engineer, you will be responsible for building and maintaining the foundational data infrastructure, including the lakehouse, data pipelines, and serving layers. Treating data as a product, you will own data contracts, monitor quality and freshness, and support internal and external analytics on our AI-powered Banking OS. The role involves working not only with tabular data but also managing a large, versioned, semi-structured documentation corpus that AI agents query through specialized tools. You will uphold engineering best practices by developing pipelines with CI/CD, automated testing, and observability, ensuring production readiness and governance.

Key Responsibilities
- Develop and maintain data pipelines extracting from diverse sources such as RDBMS, Change Data Capture, and streaming endpoints.
- Process sensitive data including observability metrics, bank accounts, and transactions, adhering to strict security and compliance standards.
- Transform raw data into business-ready analytical models within a Data Lakehouse environment.
- Ensure data consistency, quality, and governance throughout the entire data lifecycle.
- Ingest and maintain a large, versioned documentation corpus with incremental synchronization, change tracking, deduplication, and freshness management, handling mostly semi-structured or public source material.
- Build and support agent-facing MCP tools that enable AI agents to query documentation, API specifications, and release histories.
- Implement pipelines as software using GitHub Actions for CI/CD, automated testing, observability, and environment promotion across Microsoft Fabric and Azure Databricks.

Required Qualifications
- Bachelor’s or Master’s degree in Computer Science, Data Science, or a related field.
- Minimum of 5 years’ experience in data engineering, preferably with Microsoft Fabric, Azure Databricks, or Spark; experience on other major cloud platforms is acceptable with readiness to adapt.
- Strong expertise in Data Lakehouse concepts and Delta Lake technology.
- Working knowledge of data regulation and compliance, with sound judgment on when to engage legal teams.
- Proven technical leadership skills, capable of independently driving projects end-to-end and collaborating directly with stakeholders without reliance on project managers.
- Excellent written and verbal communication skills in English.
- Highly proactive, autonomous, and self-sufficient in managing projects and stakeholder interactions.
- Advanced proficiency in Python and SQL, including optimizing pipeline and query performance.

Preferred Qualifications and Benefits
- Experience designing data contracts and building platforms that enable self-service for other teams.
- Familiarity with BI tools such as Power BI in a production environment.
- Basic understanding of machine learning concepts to effectively collaborate with AI engineers.
- Knowledge of retrieval and vector search technologies, including building or maintaining vector and hybrid search layers for AI products.
- Experience preparing data for model training or fine-tuning, including anonymization of sensitive information.
- Ability to instrument retrieval processes to enable external quality measurement.

Technology Stack
- Languages: Python, SQL
- Platforms: Microsoft Fabric, Azure Databricks, Delta Lake, Azure Blob Storage, Cosmos DB, Azure Key Vault
- Processing & Orchestration: PySpark, Fabric pipelines, and notebooks
- Operations: GitHub Actions, Fabric deployment pipelines, Power BI
- AI Tools: MCP platform, vector and hybrid search technologies

This role offers the opportunity to work at the intersection of data engineering and AI, building robust, scalable data infrastructure that empowers both human and AI consumers across the organization.