Laurent Terrijn

AI Benchmarks Explained: How to Read the Scores

Cover image with the title AI Benchmarks Explained

Every new AI model launches with the same chart. Colourful bars. And surprise, surprise... their own bar is always just a little bit higher than everyone else's.

Those are benchmark scores. A benchmark is a fixed exam for AI models. Same questions, same scoring, so you can compare one model with another.

Useful? Yes. Easy to misread? Very. A model can ace the exam and still be the wrong choice for your work.

What is an AI benchmark?

A standard test. The model answers a set of questions or tasks and gets a score.

Think of a driving test.

Everyone drives the same route. Same examiner, same checklist. Pass or fail. It tells you something real: this person can drive. What it doesn't tell you is whether they can park your van in your narrow street on a Friday afternoon with a delivery truck honking behind them.

Benchmarks work exactly the same way. Some test general knowledge across lots of subjects. Some test maths. Some test coding on real software problems. Some test whether an AI can get tasks done on a computer.

Each one measures one thing. None of them measures "is this the best AI for me".

Why can a benchmark score fool you?

Three reasons. The exam gets too easy, the answers leak, or the setup quietly changes the score.

The exam gets too easy. When all the top models score in the high nineties, the test stops telling them apart. A one-point difference is just noise. People call that a saturated benchmark.

The answers leak. Models learn from huge piles of public text. If the exam questions and their answers are out there on the internet, the model may have seen them during training. Then it isn't solving the problem. It's remembering the answer sheet. That's called contamination.

The setup changes the score. How many attempts the model gets. Which tools it's allowed to use. How the question is worded. Tiny choices like these can move a score by several points. So two companies can both report the "same" benchmark... and not have run the same test at all.

Three panels: too easy because top models all score in the nineties, answers leaked into training data, and a different setup that moves the score

Three ways a benchmark score can mislead you. © Laurent Terrijn

What happened with SWE-bench Verified?

The company that helped build it stopped using it. That's all three problems in one story.

SWE-bench Verified is a set of 500 real software problems from public code projects. The model gets the problem, edits the code, and only passes if the project's own tests pass. For two years it was the coding number in almost every model launch.

In February 2026, OpenAI published why it no longer reports it. Two reasons. Some of the tests reject solutions that are actually correct. And the problems come from public projects that models are trained on, so the scores increasingly show how much of the exam a model saw during training, not how good it is at new work. Progress had also slowed down: the top score crept from 74.9% to 80.9% in six months.

OpenAI now recommends a harder, newer test called SWE-bench Pro.

The lesson isn't about coding. It's about every benchmark. Exams go stale, like bread. When you see a score, check if the exam is still fresh.

Which benchmarks are harder to cheat on?

The ones with fresh or hidden questions.

  • Questions written to resist memorising. Humanity's Last Exam is a good example: around 2,500 tough questions from experts in lots of fields, built so models can't simply have seen the answers before.

  • Questions that keep changing. Some coding tests add new problems every month, after the models have finished training.

  • Tasks that require doing, not remembering. Fixing code that must pass tests, or completing a task on a computer. You can't fake the result.

Even then, read them as a rough signal. Not a final verdict.

So how do you choose an AI model for your work?

Test it on your own work. That's the only benchmark that matches your business.

One example. Made up, not from my own business.

A small accounting office has to pick between two AI assistants. The launch charts say Model A is slightly ahead. Instead of believing the chart, they do this:

  1. Collect 20 real tasks. Ten client emails to answer. Five invoices to check. Five summaries of long documents. Remove names and private details first.

  2. Write down what a good result looks like. Before testing, so the shiny answer can't bend your opinion afterwards.

  3. Run the same tasks through both models. Same instructions, same files.

  4. Score every result: usable, needs fixing, or wrong. Let the person who normally does the work be the judge.

  5. Count the mistakes that really matter. One wrong number in an invoice check weighs more than a clumsy sentence.

  6. Check speed and price. A model that's slightly better but three times slower might not be worth it.

Two hours of work. And a result that tells you more about your office than any launch chart ever will.

Numbered steps of the 20-task test: collect real tasks, define a good result, run both models, score, count the mistakes that matter, check speed and price

The 20-task test: your own work is the benchmark that matters. © Laurent Terrijn

Quick answers

  • What is an AI benchmark?

A fixed test that gives AI models a comparable score on one specific skill.

  • What is benchmark contamination?

When test questions and answers end up in a model's training data, so it remembers instead of solves.

  • Is the model with the highest score the best?

Not for you, necessarily. Small differences are often noise, and the test may not match your work.

  • How should a business compare AI models?

Run 20 of your own real tasks through each model and score the results.

The takeaway

A benchmark tells you a model can drive. Not that it can park your van.

Look at the charts for a rough idea. Then run your own 20 tasks.

The best model is the one that makes the fewest mistakes on your work.


Further reading: AI Tokens Explained: What You're Actually Paying For

 

About Me

Laurent Terrijn in a white shirt and glasses shaking hands with a woman in an office hallway

I’m Laurent Terrijn, an entrepreneur and engineer. I see myself as a systems engineer and business builder: I build the systems businesses run on. I’m also a nerd who somehow ended up speaking ten languages. I love learning, and that curiosity pulled me deep into personal branding and artificial intelligence. On the side, explaining AI has become a passion project. I enjoy taking something that sounds complicated and making it simple enough for busy business people to understand. I do it out of curiosity, and because I like sharing what I find out. Read more on www.laurentterrijn.com

Did you enjoy this article?


Laurent Terrijn
Laurent Terrijn

I’m Laurent Terrijn, an entrepreneur and engineer. I see myself as a systems engineer and business builder: I build the systems businesses run on. I’m also a nerd who somehow ended up speaking ten languages. I love learning, and that curiosity pulled me deep into personal branding and artificial intelligence. On the side, explaining AI has become a passion project. I enjoy taking something that sounds complicated and making it simple enough for busy business people to understand. I do it out of curiosity, and because I like sharing what I find out.


Laurent Terrijn
Laurent Terrijn

I’m Laurent Terrijn, an entrepreneur and engineer. I see myself as a systems engineer and business builder: I build the systems businesses run on. On the side, explaining AI has become a passion project.

Publish0x Publish0x

Reward the author with $0.01 in crypto, and earn yourself as you read!

20% to author / 80% to me.
Rewards are FREE. Publish0x pays them, not you.

Page not displaying correctly?