Data Engineer

Job Summary

To support the development of a cutting-edge data platform, the full-time remote Data Engineer will design and operate large-scale data processing pipelines for multi-terabyte and streaming datasets, collaborating closely with machine learning engineers and researchers.

Key responsibilities

• Design, build, and operate large-scale data processing pipelines, including audio/video transcoding and feature extraction
• Deploy, scale, and troubleshoot containerized workloads on Kubernetes and AWS in production environments
• Build and maintain distributed data processing jobs using frameworks such as Spark and Ray

Required qualifications

• Hands-on experience with Kubernetes and AWS for deploying and managing containerized workloads
• Proficiency in high-performance/distributed computing frameworks like Spark and Ray
• Experience with workflow orchestration tools such as Airflow or comparable systems
• Strong programming skills in Python and SQL; experience with Golang is a plus
• Demonstrated experience building and operating large-scale data processing pipelines