How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

Seven Experiments to Find the Learning Rate

By Joshua Bolick, Awareness Software Group ~6 min read September 2026

A month ago we found something that looked important. Just before a small model suddenly gets good at its task, one direction in its training stands out. Push the model a little further along that direction and it learns sooner. Push a different model along it and nothing useful happens. We wrote that up as a special direction that belongs to the model that grew it.

This week we asked what made it special. It took seven experiments to find out, and the answer was the most ordinary thing in machine learning.

Each check removed one layer

First, it was not the direction. We built a random direction that shared just one property with the special one: how far it leaned toward where the model was already heading. It did exactly the same thing, model by model. Pushing straight along the model's own average heading did twice as much, on every model.

Bars comparing four kinds of push on how soon a small model learns — A random push with the same lean matches the special direction exactly. The model's own average heading does twice as much.

Then it was not even a push. Simply raising the learning rate, how big a step the model takes each time it is corrected, for those same 40 steps did exactly what the push did. And one fixed moment worked as well as perfect timing, so no hindsight was needed either.

Then it lost to the plainest alternative. If a short burst of faster learning helps, perhaps the model should just learn faster all the time. It should. A learning rate a quarter higher for the whole run beat the clever burst, and doubling it made the models learn 38% sooner.

Bars showing how soon models learn at four learning rates, with and without a boost — Each step up in learning rate makes the models learn sooner, and the late boost adds less and less.

The learning rate on that setup had never been tuned. It had been set low on purpose, months earlier, for an experiment about spotting the moment of learning in advance, where a slow model gives you more time to watch. Nobody changed it when the same setup was later used to measure speed-ups. The data showing how slow it was had been sitting in our own archive for a month.

Then we audited everything

If one thread could be explained by an untuned setting, others might be. So we read the code behind every speed-up we have ever published and checked which learning rate it really used. None had been tuned. For most, though, the usual setting happened to sit close to the best one, so the rate could explain only a small part of the effect. Our strongest result, a head start one model gets from another's trained parts, held up when we re-ran it at the tuned rate.

The one free speed-up, and why we do not trust it yet

Tuning properly showed a trade-off: faster rates learn sooner but end slightly less accurate. A short, early burst of the faster rate got both, the speed of the high rate and the accuracy of the low one. That is 12% less time to learn, for nothing.

Bars comparing steps to learn and final errors for three learning-rate settings — The short early boost learns as soon as the higher rate and ends as accurate as the tuned one.

Our own rules say a trick found on one setup has to be tried on another. We repeated the whole recipe from scratch on a different task and on a wider model. On the new task it did nothing. On the wider model it cost accuracy. The free speed-up was a fact about one setup.

Bars comparing the boost's effect on three setups — Sooner at no cost on the original setup; nothing on a second task; a cost in accuracy on a wider model.

What to take from it

Check the most boring explanation first. We checked it last. Each experiment tested only the next layer down, so it took seven steps to reach a setting anyone could have checked on day one.

Tune the thing you are comparing against. A trick measured against an untuned baseline can look useful simply because it partly fixes the baseline.

Look at what you already have. The answer was in our own archive, collected for a different purpose. A settings choice that makes sense for one kind of experiment can quietly undermine another kind that reuses the same setup.

And a trick that works once is a hypothesis. The one real speed-up we found did not survive a second setup, which is exactly why we tried one.

These are small models on simple tasks. The lesson about baselines applies at any scale; the specific numbers do not.

Read the direction record

Read the learning-rate record

Read the audit

Read the replication record