Technical Staff Member, Pretraining Evaluations

Job Summary

A company is looking for a Member of Technical Staff, Pretraining Evaluations.

Key Responsibilities

• Understand and evaluate tasks in the base model evaluation suite, identifying strengths and limitations
• Implement improvements to the evaluation suite by adding new tasks or removing ineffective ones
• Enhance statistical understanding and improve the signal-to-noise ratio of evaluations

Required Qualifications

• Familiarity with base model evaluations and their differences from post-trained models
• Strong statistical skills and experience with scientific experiments related to data collection and model performance
• Extremely strong software engineering skills
• Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)
• One or more papers published at top-tier venues (e.g., NeurIPS, ICML, ICLR)