Data Engineer

Job Summary

A company is looking for a Software Engineer II, Data.

Key Responsibilities

• Design and improve data pipelines for processing large, multi-modal datasets into training datasets for AI models
• Evolve the data storage layer to support analytics, schema evolution, reproducibility, and efficient data access
• Collaborate with ML engineers to enhance the performance and reliability of Python-based data processing workflows

Qualifications

• Minimum of 8 years of related experience with a Bachelor's degree; or 6 years with a Master's degree; or a PhD with 3 years experience; or equivalent experience
• Proven ability to design flexible, maintainable ETL systems
• Experience with data pipeline orchestration tools such as Prefect, Airflow, Argo, Databricks, or Spark
• Hands-on experience with multi-terabyte scale data processing
• Familiarity with AWS and knowledge of data lake technologies such as Parquet, Iceberg, AWS Glue