AI Agent Testing Specialist

Job Summary

A company is looking for an Evaluation Scenario Writer - AI Agent Testing Specialist.

Key Responsibilities

• Create structured test cases simulating complex human workflows
• Define gold-standard behavior and scoring logic for evaluating agent actions
• Analyze agent logs, failure modes, and decision paths

Required Qualifications

• 3+ years of software development experience with a strong focus on Python
• Experience with Git and code repositories
• Proficient in structured formats like JSON/YAML for scenario descriptions
• Understanding of core LLM limitations and their impact on evaluation design
• Familiarity with Docker