How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research question

What decides when a model learns, and can you change it?

By Joshua Bolick, Awareness Software Group ~4 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

Two models given the same task learn at different moments. What sets the timing, and can you move it deliberately?

Where this stands

Settings dominate, data barely matters, and there is a brief window before the jump in which interrupting the model is unusually costly. Timing can be delayed but not brought forward.

A critical window that closes as models grow is the kind of thing that matters when scheduling real training, and it is measurable here in a way it is not at scale.

Still open · 8 published results bear on this question.

What this does not settle yet

Stated plainly, because the gaps are as much a part of the record as the answers:

Nothing found so far makes the transition arrive EARLIER. Every lever delays it.
The critical window has been measured on one component; whether other components have their own windows is untested.

How these results fit together

Eight records, and they line up into a clear and slightly deflating conclusion: the timing of the jump is mostly decided by ordinary things, and it is not steerable. Ordinary training settings -- learning rate, batch size -- move the moment by hundreds of steps. Changing the entire training dataset moves it by about seven, and changing the order of the data by about three. So the exotic explanations lose to the boring ones by two orders of magnitude. We then tried to move the moment deliberately, and found the response to a nudge is discontinuous: tiny and enormous perturbations both delay learning by about the same amount, and nothing we tried made it arrive sooner. That closed a whole line of work about steering training, and it is why this project stopped trying to build a controller. Alongside that sits a genuinely causal result -- there is a window of about forty steps just before the jump where interfering with the model's memory delays it much more than interfering earlier or later. The window is real, but it narrows as models get bigger and had vanished at the largest size we tested, which is a reason to be careful about building anything on it.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.

Back to all research questions

What decides when a model learns, and can you change it?

Where this stands

What this does not settle yet

How these results fit together

The results

The Forty Steps That Matter

The Window Closes as Models Grow

We Had Been Calling It the Right Thing

Is Transition Timing Controllable?

What Actually Decides When a Model Learns

Is Learning a Lucky Accident?

Does the Effect Survive the Optimizer?

The Fix That Sharpened It

Want this measured on your data?