Senior Data Engineer

We are seeking a hands-on Senior Data Engineer to play a pivotal role in building the engineering foundation of Osol's Data & Analytics practice. This role involves standardizing data engineering approaches through proofs of concept, pilots, and reusable accelerators, applying these standards across client engagements, industrial data platforms, and the Industrial AI initiatives within our manufacturing plants. The position is platform-agnostic, working across major cloud providers, on-premises environments, or hybrid setups. You will handle diverse data ingestion scenarios, from ERP systems to plant sensors, ensuring secure and cost-efficient processing. This is a senior individual contributor role with opportunities to mentor engineers and lead a data engineering team as the practice expands.

Key Responsibilities:
- Methodology & Standards: Collaborate with the Head of Data & Analytics to define Osol’s standard data engineering methodologies, including reference patterns such as lakehouse layering, change data capture, and streaming. Establish coding and testing standards, definition of done, and estimation guidelines.
- Proofs of Concept: Plan and execute proofs of concept and pilots across cloud, on-premises, and hybrid platforms. Evaluate success based on agreed criteria and publish evidence-based comparisons covering capability, performance, cost, and operability.
- Reusable Accelerators: Develop metadata-driven ingestion frameworks for batch, CDC, streaming, and file data sources. Create data quality rule libraries, CI/CD pipelines, infrastructure-as-code templates, and demo environments to support presales and partner engagements.
- Multi-Source Ingestion: Build robust pipelines for structured, semi-structured, and unstructured data from databases, SAP and Oracle ERP systems, APIs, files, documents, logs, event streams, and IoT sensors. Support batch, micro-batch, and real-time data flows across industries such as manufacturing, finance, telecom, and retail.
- Streaming & CDC: Implement change data capture and event-driven integrations, managing challenges like late-arriving, duplicated, or out-of-order data, and backfilling extensive historical datasets.
- Quality by Design: Embed security features such as identity and access management, encryption, and data masking. Ensure governance through cataloguing, lineage, classification, and retention policies. Incorporate data quality checks, reliability measures, cost control, orchestration, automated testing, CI/CD, and SLA-based monitoring.
- Portable, Reusable Pipelines: Design collection and storage patterns that operate consistently on-premises and in the cloud. Ensure pipelines are configurable rather than recoded for each client or plant, maintaining strict data separation.
- Industrial AI Use Cases: Deliver reliable, analysis-ready datasets and feature pipelines to support use cases like predictive maintenance, transitioning pilots into stable production environments.
- Presales & Delivery: Support presales activities with technical discovery, sizing, estimates, and demonstrations. Lead technical delivery through design and code reviews, ensuring solutions are client-operable.
- Industrial Data Acquisition: Preferably, acquire data from plant historians, control, maintenance, and laboratory systems via read-only access in segmented plant networks. Manage buffering during outages and backfilling historical data without impacting plant operations.

Required Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Software Engineering, Information Technology, Engineering, or a related field.
- 7 to 10 years of experience in data engineering, data integration, or data warehousing, including at least 3 years in a technical lead or lead engineer role.
- Proven leadership in building at least one data platform or major integration program that reached production. Experience across multiple industries is highly desirable; industrial or operational technology (OT) data experience is a strong plus.
- Deep, production-grade expertise in at least one modern data platform (cloud or on-premises) and practical experience with at least one other.
- Strong programming skills in SQL, Python, and Apache Spark (PySpark/Spark SQL).
- Demonstrated ability to deliver batch and real-time pipelines using Kafka or equivalent technologies, including change data capture.
- Experience handling semi-structured and unstructured data formats such as JSON, XML, documents, and logs.
- Solid understanding of data modeling techniques including dimensional models, Data Vault, and lakehouse layer design, with the ability to translate logical models into efficient physical implementations.
- Knowledge of security, governance, and DataOps practices including access control, encryption, masking, lineage, cataloguing, automated quality checks, Git, CI/CD, infrastructure as code, and pipeline monitoring.
- Experience running proofs of concept and producing clear, evidence-based technical recommendations.

Preferred Qualifications and Benefits:
- Certifications in data engineering on major platforms such as Databricks, Microsoft Fabric, Snowflake, AWS, or Google Cloud are advantageous.
- Multi-industry experience, especially in manufacturing or industrial environments, is highly valued.
- Opportunity to work on cutting-edge Industrial AI programs and contribute to transformative manufacturing data initiatives.
- Collaborative environment with scope for mentoring and leadership growth within a rapidly expanding data practice.

This role offers a unique chance to influence the data engineering standards and solutions at the heart of industrial digital transformation. If you are passionate about building scalable, secure, and high-quality data platforms that drive real-world AI applications, we encourage you to apply.