How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Revisiting Rescues the Easy Tasks

By Joshua Bolick, Awareness Software Group ~11 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. Our hand-written practice plan for a small model moved on to the hardest task too early and let the others slip. So we added one rule, familiar from how people study: go back to any task whose accuracy has fallen below its best. The adaptive teacher it was modelled on still set the bar.

What we found. The rule closed about half the gap. The easy tasks, which had been learned and then forgotten, came back almost to the teacher's level. The middle task did not budge, because it had never been learned properly before the plan moved on, so its accuracy had nothing to fall from and the rule never noticed it.

Why it matters. The lesson applies well beyond models: revisiting what you have forgotten is cheap and worth doing, but it cannot finish what was left half-done. The teacher's remaining advantage is that it keeps practising whatever is not finished yet, whether or not it ever slipped.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, S2's six seeds plus its screening seed. The design and kill test were committed (18fbf54) and the pilot (21acf96) before any run.

Program v2 Bucket S, item S14. Decisive computation: analysis/revisiting_window.py. Output: analysis/revisiting_window.json. Reproduce with python analysis/revisiting_window.py (about forty-five minutes on a throttled laptop CPU); --reuse re-derives every endpoint from the saved curves.

The question

S8 wrote S2's frontier teacher down as a two-constant sliding window: 0.807 family accuracy against the teacher's 0.830, undecided. It parked -- reached the hardest lag halfway through and stayed -- while the middle lags decayed. The teacher's rule re-practises any lag whose running accuracy slips back toward 70%.

**A practice plan that goes back to what slipped.** The hand-written practice plan from the last experiment, the same plan with one added rule (go back to any task whose accuracy has slipped below its best), and the adaptive teacher both were modelled on. Going back to what slipped rescues the easy tasks, which had been learned and then forgotten. It does nothing for the middle one, because that task was never learned well in the first place, so there was nothing to slip. The teacher keeps practising it simply because it is not finished. Revisiting restores; it cannot complete.

The smallest rule with that property: S8's window, unchanged, plus a revisit term. Every lag the window's centre has passed gets extra weight k * (best - running) / best, where running is its accuracy on its own training batches (free, as the teacher's is) and best the highest that has reached. k was screened on S2's screening seed at 2, 8, 32.

Kill test, fixed before execution: the revisiting window's family accuracy is below the teacher's with an interval excluding zero, paired over six seeds.

Prior recorded in advance: moderate.

Anchor, in code, passed exactly: with k = 0 the rule reproduces S8's window on the first seed (0.7801).

Result: the kill test does not fire, and the gap narrows

Arm	Family accuracy	Paired
S8's window	`0.807`
revisiting window	`0.819` `[0.808, 0.830]`	`+0.012` `[-0.000, +0.025]` against the window
S2's teacher	`0.830`	revisiting minus teacher `-0.011` `[-0.046, +0.023]`

Half of S8's gap to the teacher closes, and the interval against the teacher still includes zero. One disclosure the screen forces: on the screening seed, no revisit strength beat switching the term off (0.824, 0.806, 0.810 against S8's window at 0.828), so the screen chose the least harmful setting, k = 2. On the six measurement seeds that setting helped, marginally.

Why it closes half the gap and not all of it

Final accuracy by lag:

Arm	2	3	4	6	8	10	12
window (S8)	`0.968`	`0.901`	`0.830`	`0.686`	`0.639`	`0.758`	`0.866`
revisiting	`0.982`	`0.951`	`0.896`	`0.736`	`0.634`	`0.719`	`0.815`
teacher (S2)	`0.981`	`0.958`	`0.935`	`0.848`	`0.752`	`0.682`	`0.656`

The revisit term rescues the easy lags -- 2, 3 and 4 come back to within a few points of the teacher -- and does nothing for lag 8, which stays at 0.634 against the teacher's 0.752. A revisit term can only restore what was once learned: it fires when a lag falls below its own best. Lag 8 was never learned well before the window moved past it, so its best is low and its "slip" is small, and the rule has nothing to rescue. The teacher does something different for lag 8: it keeps practising it because it is still near 70%, not because it slipped. The window still spends the second half of the run on lags 10 and 12.

What stands

The kill test does not fire: -0.011 [-0.046, +0.023] against the teacher, half of S8's gap.
A revisit rule restores what was learned and slipped; it cannot finish what was never learned. The teacher's remaining advantage is in lags that were abandoned half-learned, which is the S8 finding in sharper form.
Two constants and one line get within about a point of the teacher, and the last point needs the teacher's own criterion -- keep practising what is near the frontier -- which is no longer "a window".

Limits

The screen did not favour revisiting on its own seed; the measured gain is small and its interval touches zero. A different screen could have chosen k = 0, which is S8.
Six seeds, one task family, one width, as in S2 and S8.
The window's other constants were held at S8's values (reach 0.5), which is what parks it. A window that never outruns the frontier might need no revisit term at all; that is the teacher again.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.