Principal Engineer - Data Retrieval

Job Summary

Leading the architecture and scaling of data acquisition infrastructure, the full-time Principal Engineer - Data Retrieval will develop systems for discovering and crawling valuable data sources while ensuring high-quality data delivery to training pipelines in a remote setting.

Key responsibilities

• Design scalable frontier storage and queueing systems to support high-throughput data processing and continuous discovery loops
• Develop a shared crawl execution layer for reliable data fetching across dynamic websites while managing resource efficiency
• Implement production readiness measures, including capacity planning, observability, and fault tolerance, while coordinating dependencies across teams

Required qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience
• 15+ years of experience in software engineering with a focus on large-scale distributed systems and complex infrastructure delivery
• Deep expertise in web crawling, indexing, or ingestion infrastructure, including distributed scheduling and high-throughput data processing
• Strong engineering and system design skills with experience in building batch or streaming data pipelines
• Proven ability to deliver data platforms for demanding technical users, translating requirements into clear engineering outcomes