Data Engineer

We are looking for a skilled Data Engineer to contribute to a short-term AI application delivery project built on a comprehensive AI platform. This platform supports capabilities such as MLOps, large language models (LLM) with retrieval-augmented generation (RAG), conversational user interfaces, low-code data pipelines, and workflow orchestration. The Data Engineer will primarily focus on secure data integration and pipeline development, connecting to various client data sources, implementing and configuring pipelines, transforming and validating data, and documenting solutions for smooth operational handoff. The role also involves defining how relational, intermediate workflow, and output data are stored, structured, and made accessible to downstream applications, analytics, and AI workflows. This engagement is expected to last approximately two months, with the final solution deployed within the client’s on-premises environment.

Key Responsibilities

- Build and configure batch data pipelines using the platform’s low-code pipeline builder, supporting event- or message-based integration when necessary.
- Securely connect to diverse client data sources, including relational databases, APIs, file-based storage, on-premises systems, enterprise platforms, and messaging technologies such as Kafka.
- Develop and validate SQL queries for data extraction, transformation, filtering, aggregation, lookups, and data quality checks.
- Prepare, transform, clean, and validate data to support AI workflows, machine learning models, backend services, and frontend applications.
- Define and verify source-to-target mappings, data schemas, field mappings, data types, event/message schemas, and downstream input/output structures.
- Create and maintain YAML or configuration files for pipeline setup, data mappings, source connections, event/message integration, and workflow inputs.
- Validate data ingestion processes, including source formats, schema adherence, completeness, duplicate and null handling, retry mechanisms, event/message delivery, data freshness, and availability downstream.
- Conduct thorough data quality checks covering completeness, consistency, reconciliation, schema validation, and event/message integrity.
- Establish guidelines for storing, structuring, and making relational, intermediate workflow, event/message, and output data accessible to downstream workflows.
- Collaborate closely with Backend, Frontend, DevOps, and client IT/security teams to ensure proper data formats, secure connectivity, and successful deployment.
- Support testing, troubleshooting, deployment, documentation, and operational handoff of data pipeline components within the client’s on-premises environment.

Required Qualifications

- Proficient in SQL, including complex joins, aggregations, filtering, common table expressions (CTEs), window functions, and query optimization techniques.
- Experience in data transformation, cleansing, source-to-target mapping, and rigorous data validation.
- Hands-on experience building batch data ingestion pipelines from relational databases, APIs, file-based sources, and shared storage, including scheduling, incremental loads, retry and error handling, and delivery validation.
- Familiarity with structured and semi-structured data formats such as relational tables, JSON, CSV, Parquet, event messages, and API outputs.
- Knowledge of Kafka or similar messaging platforms, including event/message schemas and basic producer/consumer concepts.
- Experience with YAML or other configuration-driven pipeline setup methods.
- Strong background in PostgreSQL, including schema design, indexing, query optimization, and relational data modeling.
- Proven experience with secure enterprise or on-premises data integration, addressing network restrictions, credential management, and access controls.
- Excellent troubleshooting, testing, documentation, and independent delivery capabilities.

Preferred Qualifications

- Understanding of data governance principles, including access control, data retention, lineage, auditability, and secure handling of sensitive client data.
- Familiarity with Python or Java for implementing custom transformation or integration logic when required.

This role offers the opportunity to work on cutting-edge AI platform technology within a secure, enterprise environment, contributing to impactful AI-driven solutions.