Senior Data Engineer

We are seeking a hands-on Senior Data Engineer to play a pivotal role in building the engineering foundation of Osol's Data & Analytics practice. This role involves standardizing data engineering approaches through proofs of concept, pilots, and reusable accelerators, applying these standards across client engagements, industrial data platforms, and the Industrial AI initiatives within our manufacturing plants. The position is platform-agnostic, working across major cloud providers, on-premises, or hybrid environments. You will handle diverse data ingestion scenarios, from ERP systems to plant sensors, ensuring secure and cost-effective solutions. This is a senior individual contributor role with opportunities to mentor engineers and eventually lead a data engineering team as the practice expands.

Key Responsibilities

- Methodology & Standards: Collaborate with the Head of Data & Analytics to define Osol’s standard data engineering methodologies, including reference patterns like lakehouse layering, change data capture, and streaming. Establish coding and testing standards, definition of done, and estimation guidelines.

- Proofs of Concept: Plan and execute proofs of concept and pilots across cloud, on-premises, and hybrid platforms. Evaluate success against agreed criteria and publish evidence-based comparisons covering capability, performance, cost, and operability.

- Reusable Accelerators: Develop metadata-driven ingestion frameworks supporting batch, CDC, streaming, and file-based data. Create data quality rule libraries, CI/CD pipelines, infrastructure-as-code templates, and demo environments for presales and partner use.

- Multi-Source Ingestion: Build robust pipelines for structured, semi-structured, and unstructured data from various sources such as databases, SAP and Oracle ERP systems, APIs, files, documents, logs, event streams, and IoT sensors. Support batch, micro-batch, and real-time processing across industries including manufacturing, finance, telecom, and retail.

- Streaming & CDC: Implement change data capture and event-driven integrations, managing challenges like late-arriving, duplicated, and out-of-order data, as well as backfilling extensive historical data.

- Quality by Design: Integrate security features such as identity and access management, encryption, and data masking. Ensure governance through cataloging, lineage, classification, and retention policies. Embed data quality checks, reliability measures, cost control, and operate pipelines with orchestration, automated testing, CI/CD, and SLA-based monitoring.

- Portable, Reusable Pipelines: Design collection and storage patterns that operate consistently on-premises and in the cloud. Ensure pipelines are configurable rather than recoded for different plants or clients, maintaining strict data separation.

- Industrial AI Use Cases: Deliver reliable, analysis-ready datasets and feature pipelines to support use cases like predictive maintenance, transitioning pilots into dependable production environments.

- Presales & Delivery: Support presales activities with technical discovery, sizing, estimates, and demonstrations. Lead technical delivery through design and code reviews, ensuring clients receive solutions they can operate independently.

- Industrial Data Acquisition: Acquire data from plant historians, control, maintenance, and laboratory systems via read-only access in segmented plant networks. Manage buffering during outages and backfill years of historical data without disrupting plant operations.

Required Qualifications

- Bachelor’s or Master’s degree in Computer Science, Software Engineering, Information Technology, Engineering, or a related field.

- 7 to 10 years of experience in data engineering, data integration, or data warehousing, including at least 3 years in a technical lead or lead engineer role.

- Proven leadership in the technical build of at least one data platform or major integration program that reached production.

- Experience across multiple industries is highly preferred; industrial or operational technology (OT) data experience is a significant advantage.

- Deep, production-grade expertise in at least one modern data platform (cloud or on-premises) and hands-on experience with others.

- Strong programming skills in SQL, Python, and Apache Spark (PySpark/Spark SQL).

- Demonstrated ability to deliver batch and real-time pipelines using Kafka or equivalent technologies, including change data capture.

- Experience handling semi-structured and unstructured data formats such as JSON, XML, documents, and logs.

- Proficiency in data modeling techniques including dimensional models, Data Vault, and lakehouse layer design, with the ability to optimize physical designs.

- Knowledge of security, governance, and DataOps practices including access control, encryption, masking, lineage, cataloging, automated quality checks, Git, CI/CD, infrastructure as code, and pipeline monitoring.

- Capability to run technical evaluations, proofs of concept, and produce clear, evidence-based recommendations.

Preferred Qualifications and Benefits

- Certifications in data engineering on major platforms such as Databricks, Microsoft Fabric, Snowflake, AWS, or Google Cloud are advantageous.

This role offers the opportunity to work on cutting-edge industrial data projects, contributing to transformative AI use cases while shaping the future of data engineering practices within a dynamic and growing organization.