Specialist - Data Engineering

We are seeking an experienced Databricks Python Developer with 5 to 8 years of hands-on experience to join our team in Hyderabad. The ideal candidate will specialize in Databricks and Python, focusing on data engineering and reporting analytics. This role involves building, automating, and maintaining data pipelines that transform raw data into optimized reporting assets. The candidate should have strong expertise in PySpark and be comfortable working with large-scale distributed data processing environments. Experience with cloud data platforms and SQL-based data warehousing is also essential.

Key Responsibilities:
- Develop and administer solutions on the Databricks platform, ensuring optimal performance and scalability.
- Design, build, and maintain ETL/ELT pipelines within Databricks to support data transformation and reporting needs.
- Automate data processing workflows using Python and PySpark for efficient data handling.
- Extract, transform, and optimize data using advanced SQL techniques.
- Implement and manage Delta Lake and Lakehouse architecture, overseeing data lifecycle management.
- Work with a variety of data sources including structured, semi-structured, and unstructured data.
- Perform data modeling, query optimization, and performance tuning to enhance data accessibility and reliability.
- Integrate Databricks solutions with cloud platforms such as Azure, AWS, or Google Cloud Platform.
- Develop scalable and reusable data engineering frameworks to support enterprise data initiatives.
- Ensure data quality, governance, security, and monitoring practices are in place and adhered to.
- Utilize version control tools like Git and implement CI/CD pipelines to streamline development and deployment.
- Collaborate effectively with business stakeholders to translate requirements into technical solutions.
- Troubleshoot complex data issues using strong analytical and problem-solving skills.
- Communicate clearly and manage stakeholder expectations throughout project lifecycles.

Required Qualifications:
- Proven expertise in Databricks platform development and administration.
- Strong programming skills in Python, particularly for data processing automation and application development.
- Extensive experience with PySpark for distributed data processing.
- Advanced SQL skills for data extraction, transformation, and query optimization.
- Hands-on experience building and maintaining ETL/ELT pipelines in Databricks.
- Solid understanding of Delta Lake and Lakehouse architecture principles.
- Experience handling diverse data types, including structured, semi-structured, and unstructured data.
- Knowledge of data modeling, performance tuning, and query optimization techniques.
- Familiarity with cloud platform integrations (Azure, AWS, GCP).
- Proficiency in developing scalable data engineering frameworks.
- Strong grasp of data quality, governance, security, and monitoring best practices.
- Experience with version control systems (Git) and CI/CD implementation.
- Excellent analytical, problem-solving, communication, and stakeholder management skills.

Preferred Qualifications and Benefits:
- Experience with Azure Databricks and Azure Data Services such as ADF, ADLS, and Synapse.
- Hands-on knowledge of streaming data solutions like Apache Kafka.
- Familiarity with orchestration tools such as Airflow or Azure Data Factory.
- Understanding of Unity Catalog for metadata management and data governance frameworks.
- Knowledge of data warehousing concepts and modern Lakehouse implementations.
- Experience with BI and reporting tools including Power BI, Tableau, or Looker.
- Exposure to machine learning workflows and MLOps on Databricks.
- Awareness of cloud-native architecture, security, and cost optimization practices.
- Experience building reusable data products and enterprise data assets.
- Familiarity with DevOps practices, automated testing, and deployment pipelines.
- Experience working in Agile/Scrum delivery environments.
- Relevant certifications in Databricks Cloud or Data Engineering are preferred.
- Experience supporting enterprise-scale analytics and digital transformation programs is a plus.

This role offers the opportunity to work on cutting-edge data engineering projects within a collaborative and innovative environment, contributing to impactful analytics and reporting solutions.