Data Engineer  

Key Responsibilities:

1. Report to the lead for Data Management, Engineering, and Operations and will own the scope of the end-to-end development of Extract-Transform-Load pipelines for batch or real-time operations
2. Manage infrastructure under the operational scope – Decision Science and Management department. This will include working with the internal IT team to plan infrastructure capacity, install and configure applications that will be used within a data pipeline. Such examples are workflow and ETL orchestration platforms such as Apache Airflow and Talend, databases such as MongoDB, MySQL, and PostgreSQL
3. Apply software development principles such as the SDLC. This would include working with stakeholders (product managers, campaign analysts, data scientists, etc.), Project Managers, and Business Analysts to gather system requirements, build and iterate systems until they satisfy business needs.
4. Apply version control to all programming work. Code integration and Code deployment (CI/CD) for end-to-end projects should be built to manage the complexity and to own accountability of written code
5. Research and develop cost-effective ways to build data pipelines using open-source tools available
6. Serve as L3-Support for data-related issues and L2-Support for data pipelines under the scope of Data Science and Management Department
7. Write good-quality documentation of all data pipelines under their scope that will serve as a disaster recovery tool
8. Handle change-requests regarding already-implemented solutions
9. Develop in-house tools that are no available in the open-source repositories or in commercial solutions
10. Work with the Data Operations Officer to identify the root-cause of data operation issues of pipelines and will build workarounds and solutions to the issues

Qualifications
BS in Computer Science, Industrial / Management Engineering, Computer Engineering, Information Technology, or in any related field

Relevant Work Experience
3-5 years in data engineering, database management, testing, quality, and/or data governance

Required Competencies and Skills

• Expertise in Database Management (SQL, Hadoop, Hive, Ab Initio, NoSQL) and ETL
• Experienced in documentation and data quality tools
• Preferred languages: Python, SQL, Bash/Shell scripting, Java, Scala
• Knowledgeable of version control
• Stakeholder and project management
• Analytical, structure, organized, and proactive