Data Analyst  

Job Description - Overview:

The Big Data analyst provides support to the business and management with clear and insightful analyses on the data at hand. This includes data mining skills (including data auditing, aggregation, validation and reconciliation), advanced modelling techniques, testing and creating and explaining results in clear and concise reports.

Roles and Responsibilities:

• Expertise in handling other semi-structured data formats – JSON, XMLs, and HTMLs etc.
• Comfortable in querying tools for distributed systems such as Hive, Impala and other SQL clients (RDBMS, Python, Scala etc.).
• Experience in handling transforming data to suit business needs.
• Expertise in using tools for data transformation and/or ETL through tool-based and code-based development – ex. Informatica or Talend for tool based / Spark or MapReduce framework in Python or Scala/Java for code-based development
• Possesses a background in data warehousing and reporting technologies
• Designing and maintaining data systems and databases; this includes fixing coding errors and other data-related problems.
• Mining data from primary and secondary sources, then reorganizing said data in a format that can be easily read by either human or machine.
• Using statistical tools to interpret data sets, paying particular attention to trends and patterns that could be valuable for diagnostic and predictive analytics efforts.
• Demonstrating the significance of their work in the context of local, national, and global trends that impact both their organization and industry.
• Preparing reports for executive leadership that effectively communicate trends, patterns, and predictions using relevant data.
• Collaborating with programmers, engineers, and organizational leaders to identify opportunities for process improvements, recommend system modifications, and develop policies for data governance.
• Creating appropriate documentation that allows stakeholders to understand the steps of the data analysis process and duplicate or replicate the analysis if necessary.

Key Technology Requirements:

• Familiar with Data Warehousing concepts as EDS is mandated to perform key data warehousing functions
• Experienced in end-to-end DevOps techniques and be able to create Jenkins pipelines and create unit test cases for automated testing.
• Experience in Git version control.
• Experience in Docker containers and container technologies.
• Experience in Flink Streaming framework.
• Experience in using Cloud Technologies (AWS RedShift, S3 etc.)
• Comfortable in designing sustainable data architecture for Big data systems. That is balance partitioning with respect to target block sizes of the HDFS, compaction techniques, bucketing etc.
• Understand the concepts of Small Files in HDFS, Storage formats such as columnar format (Parquet and ORC) and Transmit formats (Avro).
• Familiar with resource allocation and proper resource sizing for distributed jobs. Ex. Spark executor and driver sizes

Desired Experience & Qualities:

• Strong SQL skills
• Experience in any of the following languages Python, Java, Scala
• Experience in Talend, Information, Ab Initio and other ETL GUI based tools
• Comfortable in working with Hive partitioning and bucketing

Qualifications and Accreditations:

• Amazon Web Services (AWS) Certified Data Analytics – Specialty
• Cloudera Certified Associate (CCA) Spark and Hadoop Developer