Staff Data Platform Engineer

Job Summary

Owning the end-to-end data platform for AI research, the full-time Staff Data Platform Engineer will manage continuous ingestion, preprocessing, governance, and export of high-volume media datasets, while working remotely or from the New York City headquarters.

Key responsibilities

• Oversee the data platform for AI research, ensuring reliable and reproducible datasets for image, video, audio, and text data
• Complete and scale the Databricks lakehouse and distributed processing stack, making strategic architecture decisions
• Build infrastructure for in-house synthetic data generation, enabling researchers to produce deepfake variants on demand

Required qualifications

• 8+ years of experience building data platforms or infrastructure with ownership of high-volume production pipelines
• Deep hands-on experience with distributed processing (Spark, Ray) and lakehouse platforms, particularly Databricks
• Production experience with Databricks, Unity Catalog, Delta Lake, and Spark on AWS
• Strong skills in Python and SQL, with the ability to manage complex platform challenges independently
• Experience in building reproducible ML datasets and familiarity with orchestration tools like Airflow or Dagster