Senior Machine Learning Engineer

Job Summary

To enhance the quality of AI agents, the full-time Senior Machine Learning Engineer will design, build, and operate evaluation and optimization systems for Rovo agents, working remotely or onsite in Sydney or Melbourne.

Key responsibilities

• Build offline evaluation suites and benchmarks reflecting realistic agent tasks and run regression tests before changes are deployed
• Develop online quality metrics based on production traffic, focusing on behavioral signals and model-based evaluations
• Own projects from problem definition through experimentation and deployment, contributing to design and code reviews while coaching team members

Required qualifications

• Bachelor's or master's degree in Computer Science or a related field, or equivalent practical experience
• 4+ years of industry experience in machine learning, specifically with LLM-based or agentic systems
• Strong coding skills in Python and/or Kotlin, with experience in writing production-quality code
• Experience in designing experiments and building evaluation datasets to analyze model behavior
• Proven ability to build and operate ML systems in production, focusing on quality and reliability improvements