Lead Engineer, AI Quality

Job Summary

To support the AI Quality engineering team, the full-time Lead Engineer, AI Quality will build evaluation frameworks, diagnostic infrastructure, and observability tooling to ensure the performance of AI systems while managing a growing team in a remote environment.

Key responsibilities

• Build and maintain evaluation infrastructure, including CI/CD pipelines and datasets for AI agents
• Diagnose quality issues across the agent pipeline and develop prototypes to address them
• Lead and grow the AI Quality engineering team, setting technical direction and managing daily priorities

Required qualifications

• 7+ years of experience in building and shipping production software, particularly with LLM-powered agents
• Proficiency in Ruby on Rails and familiarity with Python for data evaluation tasks
• Experience in creating evaluation or observability infrastructure for ML/AI systems
• Knowledge of designing datasets and annotation workflows for ML/AI evaluation
• Strong ability to learn quickly and adapt in a dynamic, fast-paced environment