Senior Software Engineer

Job Summary

To accelerate the development of data products, the full-time Senior Software Engineer will contribute to the architecture and improvement of a data acquisition platform, utilizing web crawling technologies and building large-scale distributed systems, all while working remotely.

Key responsibilities

• Contribute to the architecture and enhancement of data acquisition and processing platforms to improve reliability and throughput
• Develop web crawling technologies to capture and catalog data from the internet
• Design and implement backend services for managing distributed job orchestration and data pipelines

Required qualifications

• 7+ years of experience in building or operating backend or infrastructure systems at scale
• Proficient in programming languages such as Python, Go, or Rust, with knowledge of asynchronous programming
• Strong understanding of software architecture, distributed systems, and data ingestion processes
• Experience with Linux/Unix command-line tools and system resource management
• Familiarity with message queues and distributed task systems like Kafka or Airflow