Nous Research is hiring a Machine Learning Engineer, Evals

Nous Research is hiring a remote ML Engineer to own evaluation across the lab: agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties them together. Small team, high ownership — you ship eval infrastructure researchers depend on from day one.

What you'll do

• Run the full eval pipeline end to end and reproduce known results during onboarding
• Build a judge calibration protocol: measure agreement, find drift zones, document it so anyone can re-run it
• Extend benchmarks such as GAIA, tau-Bench, or a SWE-bench slice with new tasks, rubrics, and automated graders
• Run failure analysis on model outputs and turn it into recommendations for training data and judge prompts
• Own a recurring eval workflow — weekly regression suite, drift dashboard, red-team evaluation for a new capability

Requirements

• 3+ years in software, ML engineering, data science, or a research-adjacent role, with real evaluation experience
• Experience with at least one LLM evaluation framework, and opinions about where it falls short
• Hands-on LLM work: prompting, few-shot design, ideally fine-tuning or RAG
• Solid, tested Python

Remote, Americas (US time zones). Full details and application at the link.