Senior Data Platform Engineer

Job Summary

To support the Compute Platform team, the full-time Senior Data Platform Engineer will design and operate high-scale data systems for large-scale autonomous systems development, working remotely or from Pittsburgh, PA.

Key responsibilities:

• Design and operate distributed storage systems for scheduling and executing large-scale batch workloads
• Build and maintain an open-source, modern data platform while optimizing storage resource utilization
• Collaborate with cross-functional teams to understand workload requirements and enhance platform capabilities

Required qualifications:

• 7+ years of experience building and operating distributed storage systems or modern data platforms
• Experience operating streaming platforms such as Kafka or Pulsar
• Fluent in Python and SQL, with experience in highly available data applications using Trino and Apache Spark
• Knowledge of table formats like Iceberg, Delta Lake, Hudi, and Xtable
• Experience operating and optimizing at least one RDBMS (Postgres, MySQL)