Research Engineer - Web Crawlers

Job Summary

Focused on large-scale web crawling for frontier AI models, the full-time remote Research Engineer - Web Crawlers will build and operate distributed web crawlers, solve complex data extraction challenges, and design targeted crawling pipelines to create high-quality training datasets.

Key responsibilities

• Build and operate large-scale, distributed web crawlers to efficiently discover and extract data from billions of web pages
• Solve complex challenges related to content extraction, deduplication, and data freshness in web crawling
• Design targeted crawling pipelines to identify and convert high-value data sources into clean, training-ready datasets

Required qualifications

• Hands-on experience in building and scaling web crawlers or scraping systems, particularly for machine learning training data
• Strong engineering skills in distributed systems at scale, such as Kubernetes or queue-based architectures
• Ability to autonomously evaluate the quality, coverage, and compliance of crawled data
• Experience in creating tooling to measure and monitor crawled data effectively