Staff Applied Scientist

Job Summary

Leading the design and build of an evaluation framework for an enterprise planner-based agent, the full-time Staff Applied Scientist will collaborate with the client's engineering team to conduct statistically sound experiments and provide actionable insights, all while working remotely.

Key responsibilities

• Design and implement the end-to-end evaluation workflow for a planner-executor agent, including user simulation and statistical analysis
• Build a locally hosted dashboard for experiment management and insights, ensuring seamless integration with the client's platform
• Partner with client engineers to define success criteria and deliver clear technical reports that translate experimental results into business impact

Required qualifications

• Experience with LLM and agent evaluation methods, including trajectory-level scoring and user simulation
• Strong background in experimental design and statistical inference, including non-inferiority testing and power analysis
• Experience building or evaluating planner-executor architectures in production environments
• Familiarity with LLM routing, model selection, and cost-aware machine learning techniques
• Prior customer-facing or consulting experience in delivering research engagements