Principal Data Engineer

Confiz is looking for a Principal Data Engineer with over five years of hands-on experience working with modern cloud data technologies. This role is perfect for someone who is passionate about designing and implementing scalable, reliable data solutions and enjoys working in agile, collaborative teams. The position involves working primarily with Databricks and PySpark for large-scale data processing, DBT for data transformation, Astronomer/Airflow for orchestration (leveraging Cosmos to run DBT DAGs natively within Airflow), and Snowflake as the cloud data warehouse platform.

Key Responsibilities

- Build and maintain scalable data pipelines using DBT and Astronomer/Airflow to handle both batch and incremental data processing efficiently.
- Develop modular DBT transformation layers, progressing from staging to intermediate and mart layers, ensuring robust testing, comprehensive documentation, and incremental model strategies.
- Orchestrate end-to-end workflows on Astronomer/Airflow, utilizing Cosmos to trigger and manage DBT models as native Airflow DAGs.
- Design and optimize Snowflake schemas and queries, focusing on performance tuning and cost control for analytical workloads.
- Enforce data quality standards by implementing DBT tests and using pipeline observability tools to monitor data integrity.
- Write clean, well-tested Python and SQL code, actively participating in code reviews to maintain high-quality standards.
- Lead and mentor a team of data engineers, contributing to sprint planning, architectural decisions, and defining technical standards.

Required Qualifications

- Bachelor’s degree in Computer Science or a related field.
- Extensive experience with Databricks and PySpark, including Delta Lake, batch and streaming workloads, and cluster/cost optimization.
- Advanced knowledge of DBT (Core and Cloud), including data modeling patterns, incremental strategies, custom tests, and CI/CD integration.
- Proficiency with Astronomer/Airflow, including DAG authoring, using Cosmos for DBT orchestration, and pipeline observability.
- Strong expertise in Snowflake, covering data modeling, query performance tuning, access control, and cost governance.
- Solid programming skills in Python and SQL, with an emphasis on writing modular, maintainable, and testable code.
- Hands-on experience with cloud platforms such as Azure or AWS.
- Comfortable working in agile teams and familiar with CI/CD development practices.

Confiz operates a global team of talented professionals engaged in innovative enterprise projects and products. The company serves a diverse customer base, including Fortune 100 retail and consumer packaged goods companies, leading store chains, fast-growing fintech firms, and multiple Silicon Valley startups.

What sets Confiz apart is its strong commitment to process excellence and a positive workplace culture. The company holds certifications including ISO 9001:2015 (Quality Management System), ISO 27001:2022 (Information Security Management System), ISO 20000-1:2018 (IT Service Management), and ISO 14001:2015 (Environmental Management System). Confiz fosters a vibrant learning environment through collaboration and emphasizes making the workplace enjoyable.

Employees at Confiz have the opportunity to work with cutting-edge technologies while contributing to both the company’s success and their own professional growth.