Director of AI Automation Engineering

Job Summary

To enhance the quality and reliability of AI initiatives, the full-time Director of AI Automation Engineering will manage automated testing infrastructure, evaluation frameworks, and CI/CD processes while collaborating closely with AI Engineers and product teams in a remote capacity.

Key Responsibilities

• Own the infrastructure for continuous verification of autonomous agents and manage their lifecycle before production
• Create automated evaluation frameworks for LLM outputs, ensuring quality and safety benchmarks are met
• Develop release-gating infrastructure to govern agent deployment and integrate testing into CI/CD pipelines

Required Qualifications

• 5+ years of experience in building or evaluating production AI/ML systems, particularly in agentic AI or LLM-based automation
• Direct experience with evaluating LLM outputs and implementing safety and accuracy benchmarks
• Proficiency in building CI/CD pipelines with integrated test gates, such as GitHub Actions
• Experience in agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus
• Familiarity with Python and standard testing frameworks (pytest or equivalent)