AI Testing Specialist

Job Summary

A company is looking for an Evaluation Scenario Writer - AI Agent Testing Specialist.

Key Responsibilities

• Create structured test cases that simulate complex human workflows
• Define gold-standard behavior and scoring logic to evaluate agent actions
• Analyze agent logs, failure modes, and decision paths

Required Qualifications

• 3+ years of software development experience with a strong focus on Python
• Experience with Git and code repositories
• Comfortable with structured formats like JSON/YAML for scenario description
• Understanding of core LLM limitations and their impact on evaluation design
• Familiarity with Docker