Machine Learning Engineer

Job Summary

Working on a high-growth, high-ownership team, the full-time Machine Learning Engineer will manage agent capability evaluations, benchmark design, and failure analysis while developing essential evaluation infrastructure for researchers in a collaborative environment.

Key responsibilities:

• Run the full evaluation pipeline end to end and reproduce known results during onboarding
• Build a judge calibration protocol to measure agreement and document processes for reproducibility
• Conduct failure analysis on model outputs, categorizing failure modes and providing recommendations for improvements

Required qualifications:

• 3+ years in software engineering, ML engineering, data science, or a related role with evaluation experience
• Experience with at least one LLM evaluation framework and hands-on experience with LLMs
• Proficiency in Python, with a focus on writing clean, tested, version-controlled code
• Familiarity with Git, CI/CD, Docker, and the Linux command line
• Understanding of basic evaluation statistics and their implications in model assessments