AI Evaluation Engineer

Job Summary

Seeking a founding AI Evaluation Engineer, this full-time remote position will design and deploy automated testing pipelines, build evaluation infrastructure for AI agents, and implement comprehensive testing strategies for complex reasoning and multi-turn conversations.

Key responsibilities

• Design and deploy automated CI/CD testing pipelines to validate agent behavior before deployment
• Build and maintain golden datasets for measuring agent accuracy, safety, and latency
• Create adversarial test suites to ensure security against prompt injections and chaotic tool responses

Required qualifications

• Proficiency in Python or TypeScript, focusing on API testing and asynchronous systems
• Hands-on experience building LLM evaluation pipelines or statistical testing frameworks
• Expertise in validating structured outputs, including JSON schema and OpenAPI specs
• Familiarity with tracing tools such as OpenTelemetry or LangSmith
• Ability to independently design and own an evaluation architecture from scratch