Data Engineer

Job Summary

A company is looking for a Data Engineer to design and maintain data ingestion pipelines.

Key Responsibilities

• Design, build, and maintain high-volume data ingestion pipelines for processing merchant transactions
• Implement measures to ensure data quality, reliability, and optimal pipeline performance
• Share knowledge with team members and foster a culture of agile, data-driven decision-making

Required Qualifications

• Professional experience as a Data Engineer with a focus on high-volume data ingestion
• Proficiency in Python for building data pipelines
• Extensive experience with Apache Spark, including Spark SQL and PySpark
• Strong understanding of the AWS ecosystem, particularly S3 and Lambdas
• Familiarity with AWS Glue for data processing and transformations