AI benchmarks, explained

Graduate-level, Google-proof science questions (the hard 'Diamond' subset). PhD-level chemistry, biology and physics that domain experts answer at roughly 65%, designed to resist web lookup.

Around 2,500 expert-written, closed-ended questions across 100+ academic subjects. The hardest broad-knowledge exam in use; frontier models still score well below human experts.

A harder, reasoning-heavy rebuild of MMLU with 10 answer choices instead of 4, cutting saturation and prompt sensitivity across 14 disciplines.

500 human-validated real GitHub issues; the model must produce a patch that makes the repo's hidden tests pass. The standard agentic software-engineering benchmark.

Contamination-free competitive-programming problems collected continuously from LeetCode, AtCoder and Codeforces, so newer problems can't be in training data.

Research-level scientific coding: implement methods from physics, biology, chemistry and materials science papers as runnable, test-verified code.

The 2025 American Invitational Mathematics Examination — 15 hard, integer-answer competition problems. A leading discriminator of mathematical reasoning.

A 500-problem slice of the MATH dataset spanning algebra to number theory at five difficulty levels, used for step-by-step mathematical reasoning.

Real, end-to-end agentic tasks executed inside a live terminal and sandbox (build, debug, configure systems). Measures tool-use and long-horizon execution.

Tool-Agent-User benchmark: the model handles realistic customer tasks (telecom, retail, airline) by calling tools and following policy across multi-turn conversations.

Crowdsourced human-preference rating from blind, pairwise model battles on LMArena (formerly Chatbot Arena). A real-world signal of overall response quality.