KA
← All notes

How AI benchmarks are actually designed, from first principles

By

· 9 min read · AI

Every new model launch comes with a chart: a row of benchmark names, a column of percentages, and the new model's numbers in bold. For a long time I just read the bold numbers and moved on.

I'd had some ideas of my own around evaluation for a while, mostly from testing models on physics problems and noticing how differently they behaved from what the leaderboards suggested. But I went into this much deeper after coming across a job post from Epoch AI for an AI researcher. Epoch is one of the groups that takes measuring AI progress seriously, and reading about the role made me realise I couldn't clearly explain what makes a benchmark good. Not which benchmarks exist, but how you'd design one from scratch.

So instead of another list of benchmark names, this note is about the design itself. As a physicist, I find it easiest to think of a benchmark as a measuring instrument. Every instrument has to answer the same questions: what are you measuring, how do you read the result, how precise is it, and what can fool it?

1. Start with the thing you actually want to measure

Physics has a clean version of this problem. You can't measure "temperature" directly. You measure something that tracks it, like how far mercury expands in a tube, and you trust the link between the two.

Benchmarks work the same way. Nobody can measure "coding ability" or "reasoning" directly. So you pick a set of tasks where success should depend on that ability, and you measure success on those tasks instead.

This gap between the thing you care about (the construct) and the thing you can score (the proxy) is the root of almost every benchmark problem. A multiple-choice science test measures "can pick the right letter among four," which overlaps with "understands science" but is not the same thing. The first design question is always: if a model scores high here, what am I actually allowed to conclude?

2. Choose the tasks: a sample from a much bigger world

A benchmark is never the full skill. It's a sample. If you want to know how good a model is at fixing bugs in real software, you can't test it on every bug that will ever exist. You collect, say, 500 real bugs and hope they're representative.

That means the way you collect tasks decides what you can claim:

  • Where do tasks come from? Written fresh by experts, taken from exams, scraped from real work (like GitHub issues), or generated by a program.
  • How varied are they? 500 bugs from just three projects measure something much narrower than 500 bugs from 100 projects.
  • How hard should they be? This is subtle. If every task is easy, all good models score 100% and you learn nothing. If every task is impossible, they all score 0% and you also learn nothing. The useful information lives in tasks that some models solve and others don't.

That third point is why good benchmarks deliberately spread difficulty from easy to very hard. Epoch's FrontierMath, for example, runs from hard undergraduate problems up to research-level ones that take expert mathematicians hours or days. A spread like that keeps the instrument useful as models improve, the same way a good ruler has both millimetre and centimetre marks.

There's also a trap here. Some benchmarks keep only the questions that current top models get wrong (Humanity's Last Exam was built partly this way). That makes the benchmark hard on day one, but it also means scores are artificially low at launch and then jump quickly, partly because models were selected against rather than because they suddenly got much smarter.

3. Decide how to score: the "ground truth" problem

Once a model gives an answer, how do you decide if it's right? There are only a few basic options, and each has trade-offs.

Exact answers. The answer is a number, a letter, or an expression that can be checked automatically. Cheap, objective, and repeatable. This is why so many maths benchmarks ask for a final number. FrontierMath goes further and designs problems so the answer is hard to guess, like a large specific number rather than "yes" or "no."

Executable checks. For code, run it against tests. SWE-bench does this with real projects: the fix counts only if the project's own tests pass. It's objective, but only as good as the tests. Weak tests let wrong solutions slip through.

Rubrics and judges. For open-ended work like essays, analyses, or plans, there is no single right answer. Someone has to grade it, either human experts with a rubric or another AI model acting as a judge. More realistic, but also more expensive, slower, and less consistent. An AI judge can also share the blind spots of the model being judged.

Human preference. Show people two answers and ask which is better, like LMArena does. With enough votes, you can turn pairwise comparisons into a single rating, the same way chess ratings work. This captures what people actually like, but people also reward confidence, length, and nice formatting, which aren't the same as correctness.

What I found most interesting: the more realistic the task, the harder it is to score objectively. Benchmark designers are always trading realism against reliability of grading.

4. Treat the score as a measurement with error bars

This is the part I think most launch charts ignore, and it's pure statistics.

If a benchmark has $n$ questions and a model gets a fraction $p$ right, the uncertainty in that score (the standard error) is roughly

$$\text{SE} \approx \sqrt{\frac{p(1-p)}{n}}$$

Take a benchmark with about 200 questions and a model scoring 70%:

$$\sqrt{\frac{0.7 \times 0.3}{200}} \approx 0.032$$

That's about ±3 percentage points for one standard error, and closer to ±6 for a 95% confidence range. So when one model scores 71% and another 73% on a 200-question test, that difference is basically noise. Small benchmarks can only tell apart models that are far apart.

There's a second source of noise: the model itself. Ask the same question twice and you can get different answers. So designers also have to decide:

  • How many attempts? Score only the first try, or count it as solved if any of several attempts is right ("pass@k")?
  • How to combine attempts? Majority vote across several runs often scores higher than a single run.

Two labs reporting the "same" benchmark with different choices here aren't really reporting the same number.

5. Control how the test is run

In physics, you'd never compare two measurements taken under different conditions without saying so. In AI benchmarks, conditions vary a lot:

  • The prompt: exact wording and examples given can shift scores noticeably.
  • Tools: can the model search the web, run code, use a calculator?
  • Thinking budget: reasoning models score higher when allowed to think longer.
  • The scaffold: for agent benchmarks, the surrounding software (how the model sees files, how many steps it gets, what happens when it errs) matters as much as the model.

So a benchmark score is really a score for a model plus a setup. This is a big reason independent evaluators matter: they run every model under the same, documented conditions, using open evaluation frameworks such as Inspect, so the comparisons are actually fair. Good evaluations also report the cost of each run, because a slightly higher score at ten times the price isn't the same achievement.

6. Protect the test from leaking

A benchmark only works if the model hasn't seen the answers. But models are trained on huge amounts of internet text, and benchmarks tend to end up on the internet. When that happens, the score starts measuring memory instead of ability. This is called contamination.

The main defences, from first principles:

  • Keep tasks private. Release only a few examples and keep the real test set hidden. FrontierMath keeps most problems private for this reason.
  • Use fresh tasks. Draw from problems created after a model's training data was collected, like each year's new maths competition, or continuously updated question sets.
  • Mark the data. Some benchmarks embed a unique "canary" string so labs can filter them out of training data, and so researchers can test whether a model has seen it.
  • Check for memorisation. If a model can complete a benchmark question word for word from a partial prompt, that's a warning sign.

7. Plan for the benchmark's death

Every benchmark eventually saturates: the best models score close to the maximum, and it stops separating them. Often the last few percent can't even be reached, because some questions have mistakes in them. If 3% of answers in the key are wrong, 97% is effectively a perfect score.

There's also Goodhart's law: when a measure becomes a target, it stops being a good measure. Once a benchmark becomes famous, labs optimise for it, sometimes in ways that improve the number more than the skill.

Good designers expect this and build in headroom: tasks well beyond current ability, private holdout sets, and versions that can be updated.

8. Connect benchmarks across time

Here's a problem I hadn't thought about before. Each benchmark lives only a few years. So how do you compare a model from 2021 with one from 2026, when no single benchmark covers both?

The solution borrows from how standardised exams compare students across different test papers, a method called item response theory. The idea: every model has an underlying ability, every benchmark (or question) has a difficulty, and the chance of success depends on the gap between the two. If you have enough overlapping results, with some models tested on both old and new benchmarks, you can estimate abilities and difficulties together on one shared scale.

That's essentially what Epoch's Capabilities Index does: stitch many benchmarks together into one long ruler, so progress can be tracked across years even as individual benchmarks saturate. I loved this, because it's exactly how physicists calibrate instruments that each work only over a limited range.

My own ideas, revisited

Thinking about benchmarks this way made the ideas I'd had before more concrete:

  • Consistency deserves its own number. Ask a model the same physics question five times and you can get different answers. A single score hides that. Reporting how often a model is reliably right across repeated runs would tell you more about whether you can trust it.
  • Verifiable research problems. In my field, many derivations end in a checkable result, like the maximum efficiency of an energy extraction process for a given black hole spin. A set of these could test real derivation ability with exact, automatic grading, in the same spirit as FrontierMath.
  • Cost per correct answer. For real use, accuracy per dollar often matters more than raw accuracy.

Some of these already exist in some form, which was humbling and also reassuring. It meant I was thinking in the right direction.

A checklist I now use

When I see a new benchmark result, I ask:

  1. What exactly is being measured, and what am I allowed to conclude from it?
  2. How many tasks? Is the difference between models bigger than the error bars?
  3. How is it graded? Exact answers, tests, rubric, AI judge, or human votes?
  4. Under what conditions? Tools, attempts, thinking time, scaffold, cost.
  5. Could it be contaminated? Public or private, old or fresh?
  6. Is it saturated? Are all top models bunched near the ceiling?
  7. Who ran it? The lab itself, or an independent evaluator?

Benchmarks are imperfect, but they're the closest thing we have to a shared ruler for AI progress. And designing a good ruler turns out to be one of the most interesting problems in the field.