Data Engineer

We are seeking a seasoned Data Engineer with extensive experience in Big Data technologies and distributed data systems. The ideal candidate will have a strong background in Hadoop, Spark, Hive, Java, Python, and advanced SQL. This role requires hands-on expertise in developing and optimizing data processing workflows, ensuring data quality, and leading technical deliverables. The successful candidate will be responsible for designing scalable data solutions and collaborating with cross-functional teams to meet business objectives.

Key Responsibilities

- Design, develop, and optimize Big Data solutions leveraging Hadoop, Spark, Hive, and related technologies.
- Develop and fine-tune complex Java, MapReduce, and Hive jobs to maximize data processing efficiency.
- Lead backend and distributed data system initiatives while maintaining hands-on involvement in technical implementation.
- Analyze and validate source data to guarantee data integrity and accuracy.
- Design and implement dimensional data models to enhance data accessibility, efficiency, and quality.
- Build and maintain stream-processing solutions using Spark Streaming, Storm, or similar frameworks.
- Write and optimize advanced SQL queries, including database performance tuning.
- Support deployment and auditing of data models and attributes to maintain accuracy.
- Manage technical deliverables and coordinate with relevant teams to prioritize projects aligned with business needs.
- Collaborate with team members and stakeholders to troubleshoot and resolve complex data, software, and technical challenges.

Required Qualifications

- 7 to 10 years of experience in Big Data Engineering or a closely related field.
- Deep knowledge of Hadoop, Cloudera, Spark, and associated Big Data frameworks.
- Hands-on experience with Hadoop ecosystem components such as HDFS, MapReduce, Pig, Hive, HBase, Flume, and Sqoop.
- Strong programming and scripting skills in Java, Python, Scala, or similar languages.
- Proficiency with Big Data querying tools like Hive and Impala.
- Advanced SQL expertise, including query optimization and performance tuning.
- Experience working with both relational and NoSQL databases, particularly PostgreSQL and Cassandra.
- Familiarity with stream-processing technologies such as Spark Streaming and Storm.
- Solid understanding of Linux environments and the ability to troubleshoot complex data, networking, and software issues.
- Excellent planning, organizational, communication, and interpersonal skills.
- Demonstrated ability to take ownership of deliverables and collaborate effectively within a team setting.

Preferred Qualifications and Additional Benefits

- Experience with Azure cloud services is advantageous.
- Knowledge of data science and machine learning technologies is a plus.
- Background in the financial services domain will be considered an asset.

This role offers the opportunity to work on cutting-edge Big Data projects within a collaborative environment, contributing to impactful data solutions that drive business success.