facebook Data Engineer

Data Engineer

aws(103726)

5-15 Years

Full Time Jobs

Market Rate

Malton, UK

#Python#Spark#Hive#Hdfs#Impala#Sqoop#Informatica

Job Description:

Responsibilities

  • Participate in requirement gathering, analysis, and solution design to build big data engineering applications on AWS Data services such as AWS EMR, and Snowflake using Spark and AWS Glue as ETL framework.
  • Migrate and build Cloudera-based Hadoop, hive, impala, Kafka, Sqoop, Spark-based data, jobs and security policies to AWS EMR, S3, Snowflake and other AWS services such AWS Glue, DMS, and IAM.
  • Hands-on involvement in Low-Level Design, Development and architecting of large
  • Data projects leading developer and testing teams.
  • Job Scheduling and Automation.
  • Data Validation, Quality checks, profiling and data reconciliation testing.
  • Work as an individual contributor as well as lead teams when required.
  • Mentor junior members of the team by improving their skills and knowledge and can get things delivered.
  • Work with both senior and junior team members like Project Manager, Hadoop
  • Architect, other data engineers, data modellers, report developers, and testers to complete the task.
  • Setting up security and governance policies on data, users, and data pipelines on AWS data services.
  • Troubleshooting application errors and ensuring that they do not occur again.
  • Apply agile and CI/CD methodologies and tools for development and deployment in agile mode.
  • Must have: Streaming Data Engineering and Analytics over it. Kafka, AWS Data Services, AWS, ETL, Spark/Scala, Java, Python, EMR, AWS Glue, AWS Athena
  • Good to have: Python, Spark, Hive, HDFS, Impala, Sqoop. Informatica

Qualification -

  • Bachelor’s degree and 9+ years of experience in the IT industry.
  • 5+ years’ experience as a Big Data Engineer on Hadoop, AWS EMR and its ecosystems
  • Experience in Banking Domain Plus.
  • Experience on AWS EMR, AWS Glue, Athena, S3, DMS, SCT and Cloudera CDH-based Hadoop.
  • Should have extensively worked on Big Data projects on AWS EMR-based Hadoop, HDFS/S3, Spark, Hive, and Impala leading teams and interacting with Architect roles and Clients.
  • Work experience with SQL, RDBMS, and complex queries.
  • Understanding and experience in data warehousing data modelling concepts.
  • Has an understanding of large Batch and Stream processing.
  • Knowledge of Quality Assurance methodologies, and exposure to all facets of
  • Extract, Transform and Load (ETL) processes.
  • Proficiency in Linux / Unix command line.
  • Experience in applying agile and CI/CD tools and methodologies for deployment and related automation.
  • Experience in performance optimization techniques on Hadoop, HDFS, Hive, Spark, and File Formats, and providing technical guidance to other application developers.


Work location: Malton, UK

Contract Duration: 12 Months


Trending Jobs

Program Manager (Banking Domain) (103728)

Malton, UK

AWS Developer (93197)

Netherlands