How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research practice

AI Models Have Critical Periods Too

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

Biologists have a term for a stretch of time in which something has to happen or development goes wrong: a critical period. A kitten's visual system needs light during a specific window early in life. Miss it and the wiring does not form properly, and providing the light later does not fix it.

We went looking for the equivalent in a small AI model, and found one.

The setup

The models we study do not improve smoothly. They sit at a low score for a long stretch and then suddenly get the task right. Something must be getting built during the stretch before that jump.

A model has distinct parts: a piece that reads the input, a memory that carries information forward, and a readout that turns the internal state into an answer. We wanted to know which part has to be working, and when.

So we stopped one part at a time from learning, for forty steps, and then let it resume. Then we measured how much later the model got there.

Almost nothing cares when you interrupt it

Paired bars showing that only the recurrent part is sensitive to when it is paused — Each part was paused twice: once immediately before the moment of learning (pink) and once at a random earlier point (grey). If a part has a critical period, its pink bar should be taller than its grey one.

For three of the four parts, the two bars match. Pausing them costs the same wherever you do it, which means they have no special window.

Look at the rightmost pair. That is the model's output layer, and both bars are exactly zero. You can switch off the part that produces the actual answers for forty steps and the timing of learning does not notice. This lined up with a separate experiment of ours that had reached the same conclusion by a completely different route, which is the sort of agreement that makes a result more believable.

The memory is the exception. Pausing it just before the model learns costs more than twice as much as pausing it earlier.

The check that made it a real finding

There was a problem with that comparison, and it is the kind that is easy to miss. The pause placed "just before the model learns" is also, necessarily, the later of the two pauses. So a part whose interruptions simply get more expensive as training goes on would produce exactly the result we saw, with nothing special about the moment of learning at all.

Those two possibilities look identical in the first figure and are completely different claims. Separating them meant sliding the same forty-step pause along the training run and measuring the cost at each position.

Line chart showing the cost peaking at the moment of learning and collapsing after — The same pause, slid along the run. Zero on the horizontal axis means the pause ended exactly when the model learned. A cost that simply grew with time would still be rising at the right-hand edge.

The cost climbs as the pause approaches the moment of learning, peaks right at it, and then falls off a cliff. Forty steps after the model has learned, pausing the same part for the same forty steps costs essentially nothing.

So the memory goes from being the most timing-sensitive part of the model to completely irrelevant, in the space of forty steps. That is a critical period, and it is a narrow one.

In plain English

Think of building an arch. There is a window during construction when the supporting frame is holding everything up and removing it would collapse the whole thing. Before that, there is not much to collapse. After the keystone goes in, the arch holds itself up and the frame can be taken away without anyone noticing.

The model's memory is that frame, and the moment of learning is the keystone going in.

Nothing was broken

Worth being clear about, because it is the part that surprised us: every version of the model still learned the task, to the same accuracy, in every single run.

We changed when learning happened. We never changed whether it could.

That is a meaningfully different result from "this part is essential". The model is more robust than the critical-period framing might suggest. Interrupt the important part at the important moment and it recovers, just later. Nothing here is a single point of failure.

Why we bothered with the second figure

The honest answer is that the first figure alone would have been publishable and wrong, or at least unsupported. It showed a real difference, and a real difference is consistent with two explanations that mean opposite things.

The extra experiment cost about half an hour of laptop time. This is the third time in as many weeks that the deciding factor in one of our results was a control run rather than the original measurement, and the pattern is getting hard to ignore: the interesting question is usually not "is the effect real" but "is the effect the thing I think it is".

The full record, with every number and the limits