Senior AI Data Engineer

We are looking for a Senior AI Data Engineer to lead the design and operation of the data infrastructure that supports our AI systems. This role is responsible for managing the flow, modeling, and quality of data from source systems through the data warehouse to the retrieval and feature layers that enable large language model pipelines, agentic workflows, and analytical products. The ideal candidate is primarily a skilled software engineer who approaches data engineering with rigor—building well-modeled warehouses, writing maintainable production Python code, and treating data pipelines as versioned, tested, and observable software rather than simple scripts. This position works closely with AI/ML Data Scientists, who focus on model behavior and retrieval strategies. The key distinction is that you will own the pipeline, schema, and data guarantees, while they own the algorithms, prompts, and evaluation.

Key Responsibilities

Data Warehousing & Dimensional Modeling
Design and evolve dimensional models using Kimball methodology, including star schemas, conformed dimensions, and enterprise bus matrices. Implement fact tables such as transaction, periodic snapshot, and accumulate snapshot with clear grain definitions. Manage slowly changing dimensions (Types 1, 2, 3, and hybrids) with proper effective dating and surrogate key strategies. Model semi-structured and unstructured data sources—documents, transcripts, event streams—into queryable formats without losing provenance. Maintain a governed semantic layer ensuring consistent metric definitions for both human analysts and automated consumers.

Pipeline & ETL/ELT Engineering
Build and maintain batch and streaming pipelines using orchestration frameworks like Airflow, Prefect, or Dagster, supporting backfill, replay, and idempotent retries. Develop ELT transformation layers with tested, documented, and version-controlled SQL. Own data contracts between producers and consumers, managing schema evolution and compatibility. Instrument pipelines for observability, including freshness, volume, distribution, and schema-drift monitoring, with alerting that differentiates real incidents from normal variance. Maintain end-to-end data lineage from source to warehouse to retrieval.

Data Engineering for AI
Develop embedding generation and refresh pipelines, including chunk materialization, embedding jobs, incremental re-embedding, and index lifecycle management. Operate vector stores such as Pinecone, Weaviate, Chroma, Milvus, or pgvector as production systems, managing capacity, index strategies, and staleness SLAs. Build extraction pipelines over unstructured data using OCR, document parsers, and vision-language models, treating extraction confidence as a key attribute. Implement preprocessing and feature pipelines that support model training and inference, ensuring train/serve consistency. Expose data to AI services through well-defined FastAPI or gRPC interfaces.

Engineering Craft
Apply SOLID principles and domain-driven design, creating bounded contexts aligned with business domains and a ubiquitous language shared with stakeholders. Maintain comprehensive test coverage including unit, contract, and data-quality assertions, treating untested pipelines as incomplete. Own CI/CD pipelines, containerization, and environment promotion for data services. Contribute to code reviews, architectural decisions, and internal standards.

Governance & Cost Management
Implement access control, PII handling, retention, and audit requirements at the data layer. Manage warehouse and pipeline costs through partitioning, clustering, materialization strategies, and storage tiering.

Collaboration
Translate business problems into data models in partnership with product and client stakeholders. Mentor junior and mid-level data engineers to build team capability.

Required Qualifications
5-10+ years of experience in software or data engineering with extensive production data platform work. Demonstrated expertise in Kimball dimensional modeling, including grain selection, many-to-many relationship resolution, and denormalization judgment. Expert-level SQL skills, including window functions, CTEs, query plan analysis, and performance tuning on columnar warehouses. Production-grade Python proficiency with typing, packaging, dependency management, and testing. Strong understanding of SOLID principles and domain-driven design with practical examples. Experience with orchestration tools such as Airflow, Prefect, or Dagster in production environments. Expert knowledge of cloud platforms (AWS, Azure, or GCP) covering storage, compute, IAM, networking, and cost management. Familiarity with containerization, Kubernetes (EKS/AKS/GKE), and CI/CD pipelines. Experience with Lakehouse table formats like Iceberg, Delta Lake, or Hudi, including compaction, snapshot expiry, and schema evolution.

Preferred Qualifications
Experience building data layers for production retrieval-augmented generation (RAG) systems, including hybrid search infrastructure and index freshness guarantees. Familiarity with streaming technologies such as Kafka, Kinesis, Flink, or Spark Structured Streaming. Knowledge of dbt or similar transformation and testing frameworks. Experience with data quality tools (Great Expectations, Soda) and data catalog/lineage platforms. Understanding of ML lifecycle tools like MLflow, Weights & Biases, and feature stores to collaborate effectively with data scientists. Proficiency in a second programming language such as TypeScript, Java, Go, Scala, or Rust. Awareness of AI security, governance, and compliance frameworks. Contributions to open-source data or AI infrastructure projects are a plus.