AI Agent Evaluation Analyst

Job Summary

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

• Review evaluation tasks and scenarios for logic, completeness, and realism
• Identify inconsistencies, missing assumptions, or unclear decision points
• Help define clear expected behaviors (gold standards) for AI agents

Required Qualifications

• Excellent analytical thinking about complex systems and logical implications
• Familiarity with structured data formats (ability to read JSON/YAML)
• Experience with policy evaluation, logic puzzles, or structured scenario design
• Background in consulting, academia, or research
• Some understanding of scoring or evaluation in agent testing