A Model Can Ace Its Own Test and Still Slow You Down

We run a research programme on models small enough that every experiment can be rerun on a laptop in minutes. It exists so that what we tell clients rests on a process we can show you rather than on results we assert.

Here is a result from it with an unusually direct practical edge, because the question it answers is one almost every AI project has to answer early: which existing model should we build on?

The setup

You can start a new model from a piece of an already-trained one instead of from scratch. When it works it removes a large chunk of the training. When it goes wrong, and it does go wrong, the new model ends up slower than if you had never borrowed anything.

So the question is how to tell in advance. We have a check for that. It costs one pass of the data, which is effectively free next to a training run, and it scores each candidate on how much it already knows about the job you actually want done.

This experiment put nine candidate models in a pool and asked what the check is worth as the pool gets bigger.

The row that stopped us

One candidate in the pool had learned its own task perfectly. A score of 1.000. Nothing left to get wrong.

Building a new model on it made that model slower, by about nine steps of training, than starting from scratch would have. It was a net cost, and it was a net cost while holding a flawless score on its own benchmark.

The free check caught it. It scored that candidate near the bottom of the pool, for one pass of the data, without anyone training anything.

Why did a perfect model hurt? Because it was excellent at a task that was almost the right one. Close enough to look ideal on paper, different enough that the new model had to spend time undoing what it had been handed. Being nearly right is its own failure mode, and it is the one a benchmark score is worst at catching.

The other half: the check has a ceiling

The check earns its keep, and it stops earning quickly.

With three candidates to choose between, it is worth about twenty steps. With six, it is worth about twenty steps. The line is flat. Meanwhile the best option in the pool keeps getting better as the pool grows, because a bigger pool contains better options, so the gap between what the check picks and what was actually available widens from about nine percent to about twenty one percent of the available benefit.

We wrote that prediction down before running the experiment, along with the reason. The check is excellent at separating helpful candidates from harmful ones and only mediocre at ranking the helpful ones against each other. Avoiding a disaster is what you need when you have three options. Picking a winner is what you need when you have thirty, and that is a different instrument.

What to take from this

Screen your candidates, do not rank them. A cheap check is worth running to throw out the choices that would cost you, and it is not worth trusting to find the single best one. Those are different jobs and the same tool does not do both.

A larger shortlist is worth less than it looks. If the way you choose saturates at three candidates, assembling thirty is mostly work you will not be paid for. Assembling thirty is also the expensive part: each candidate is a trained model.

Do not choose a base model on its benchmark score alone. The score answers "is this model good", and you are asking "is this model good for what I am about to do". Test the second question directly. On a real project that means a small, cheap probe on your own data before committing to a base model, not after.

The honest caveats

Two, because the argument this whole site makes is that you should check rather than take our word for it.

• The headline version of this experiment is confounded and we say so in the record. If you leave near-identical candidates in the pool, the choice becomes trivial and the check looks perfect. The numbers above come from the version with those removed, which is the honest one. Both were written into the code before we ran it.
• These are small models on synthetic tasks. The mechanism is what travels, not the step counts. "A perfect score does not mean a useful starting point" is a statement about what benchmarks measure, and that does not get less true at scale.

Read the full technical record

Get new results as we publish them

Roughly monthly, one finding per email, in plain English first. Including the approaches that turned out not to work, which are usually the useful ones. No sales email.

Leave this empty Email address Subscribe

Double opt-in: we send a confirmation link and add nobody who does not click it. One-click unsubscribe on every email.