Data Engineer

Purpose :

Looking for a prospect candidate who creates and manages a company’s Data infrastructure and tools, and is someone that knows how to get results from vast amounts of data quickly.

The actual definition of this role varies, and often mixes with the Data Scientist role. Here, we will assume that it is a role focused on engineering, without statistics and strong machine learning skills required.

Essential Reponsibilities:

The day to day Duties and Responsibilities of the Position are as follows:

1.       Design, construct, install, test and maintain highly scalable data management systems

2.       Ensure systems meet business requirements and industry practices

3.       Build high-performance algorithms, prototypes, predictive models and proof of concepts

4.       Research opportunities for data acquisition and new uses for existing data

5.       Develop data set processes for data modeling, mining and production

6.       Integrate new data management technologies and software engineering tools into existing structures

7.       Create custom software components (e.g. specialized UDFs) and analytics applications

8.       Employ a variety of languages and tools (e.g. scripting languages) to marry systems together

9.       Install and update disaster recovery procedures

10.   Recommend ways to improve data reliability, efficiency and quality

11.   Collaborate with data architects, modelers and IT team members on project goals

12.   Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, write SQL queries, etc.).

13.   Work closely with our engineering team to integrate your amazing innovations and algorithms into our production systems.

14.   Process unstructured data into a form suitable for analysis – and then do the analysis.

15.   Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities

16.   Implementing ETL process

17.   Monitoring performance and advising any necessary infrastructure changes

18.   Defining data retention policies

19.   Data Engineer will be involved in a number of development projects, implementation projects or support requests at any given time. You will work with Team Leads, Business Analysts and System Support staff as directed by their Software Architect.

20.   Develops software solutions by studying information needs; conferring with users; studying systems flow, data usage, and work processes; investigating problem areas; following the software development lifecycle.

21.   Determines operational feasibility by evaluating analysis, problem definition, requirements, solution development, and proposed solutions.

22.   Prepares and installs solutions by determining and designing system specifications, standards, and programming.

23.   Work at the direction of the Development Team Lead OR Software Architect from a development practices perspective. This includes but is not limited to following:

a.       Design/Architectural guidelines

b.      Applicable software coding conventions

c.       Source/Configuration management procedures

24.   Provide inputs on the technical architecture and design

25.   Coordination with peers and Lead Data Engineers for accurate and recommended development approaches for assigned tasks.

26.   Your Software Architect will ensure that you are aware of which projects/requests are precedence priority at all times. If, at any time, priorities are unclear then you should talk to your Software Architect to resolve any issues. 

Experience

●     Minimum Experience – 1 year of professional experience as Data Engineer

●     Maximum Experience – 3 year of professional experience as Data Engineer

A candidate for the position will have as at least 3 years of working experience as a database engineering support personnel or a database engineering administrator within a fast-paced a complex business setting. The candidate will also have had experience working with MySQL databases and Microsoft SQL.

The candidate must demonstrate experience working with large and complex data sets as well as experience analyzing volumes of data through basic Microsoft Excel functions, for example, macros and pivot tables. A candidate for this position will also have had experience working in internet technologies, for example, SaaS, IaaS, and PaaS.

A suitable candidate will also have had experience in the creation and debugging of databases critical to the business’s mission. The candidate will have strong working and conceptual knowledge of building and maintaining physical and logical data models and experience with Tableau, Domo or other business intelligence tools.

A suitable candidate for the position will also have had system management expertise with monitoring, disaster recovery, backup, automated testing, automated schema migration, and continuous deployment.

Supervisory Responsibilities

●     Supports and guide Data Engineers by providing advice, coaching and suggestions.

Communication Skills

●     Ability to effectively communication information and respond to questions in person-to-person and small group situations within Teams and other employees of the organization.

●     Able to understand and respond back to written emails, tasks and reports in English

Communication skills for the Data Engineer are non-negotiable. Communication skills will be needed in his managerial role where he has to convey messages and instructions clearly to the supporting personnel in order to ensure efficient execution of duties within the junior department.

Communication skills will also be necessary in his collaborative capacity. Due to the technical nature of the Data Engineer's role, he will be required to have excellent communication skills enabling him to convey technical messages to collaborative non-technical departments and personnel, leading to a positive outcome of the Data Engineers partnerships and the overall improvement in business performance.

The Data Engineer will also be required to draft regular performance and progress reports and prepare presentation for senior data engineering management and senior data science leadership, reports that have to be clear, concise, engaging, and convincing, which will require exceptional communication skills to deliver.

Technological Savvy/Analytical Skills

The Data Engineer must also have exceptional analytical skills, showing fluency in the use of tools such as MySQL and strong Python, Shell, Java, PHP, and T-SQL programming skills. He must also be technologically adept, demonstrating strong computer skills. The candidate must additionally be capable of developing databases using SSIS packages, T-SQL, MSSQL, and MySQL scripts.

The candidate will also have an ability to design, build, and maintain the business’s ETL pipeline and data warehouse. The candidate will also demonstrate expertise in data modeling and query performance tuning on SQL Server, MySQL, Redshift, Postgres or similar platforms.

Interpersonal Skills:

A candidate for the position must also possess certain personal attributes that make him more suited for the position. The candidate must be result-driven, be an analytical and creative thinker, be an innovative problem solver, be self-motivated and proactive, be highly organized, have ability to handle-multiple and simultaneous tasks meeting aggressive deadlines, be a team player, and demonstrate an exceptional ability to stay calm and composed in the face of adversity.

People Skills:

People skills like communication skills are a necessity for this position. The Data Engineer will be an approachable and relatable individual who forms strong connections with others and inspires trust and confidence in both his juniors and seniors.

This will enable junior support personnel to follow more easily in his directives and for senior management and leadership to give credit and consideration to his insights and judgments, enabling smooth performance of his duties.

Mathematical Skills

●     Ability to work with advanced mathematical equations and logics.

Critical Thinking Skills

●     Ability to solve practical problems and deal with a variety of known variables in situations where only limited standardization exists.

●     Ability to interpret a variety of instructions furnished in written, oral or diagram formats.

Planning

●     Considerable responsibility with regards to general assignment in planning time, method, manner and sequence of performance of own work.

Analytical Ability / Problem Solving

●     Directed, Supervisory and Professional skills using structured practices or policies and directed as to execution and review.  Interpolation of learned things in moderately varied situations where reasoning and decision-making are essential.