How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 44 of 45

We Tried to Spend Our Own Result

By Joshua Bolick, Awareness Software Group ~14 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. This project studies how AI models learn, and it exists to find ways of making training cheaper. Until this experiment everything we had done was measurement: we had never once taken one of our own findings and tried to actually save anything with it, which is a slightly embarrassing thing to admit about a project with that stated purpose. The finding we tried to spend is our best one. We can read a model's internal state and tell that it is about to get much better at its task, before its answers improve. The obvious way to profit from a warning like that is to ease off once you know the model is nearly there, so at the moment the warning fired we cut the amount of data used per training step by four, which makes every step cheaper.

What we found. We gave every version the same total budget so the comparison was fair, and we also ran a version that made exactly the same cut at a moment chosen at random. The cut helped: both versions reached the accuracy target on about a fifth less data than ordinary training. But the version timed by our warning did no better than the version timed at random, so the warning itself, the entire point of the exercise, contributed nothing. Then we added up something we had built into the experiment but had not expected to decide it. To act on a warning you have to keep checking for it, and checking is not free. Watching for the moment consumed more than half the whole training budget, which is roughly thirteen times what the manoeuvre saved, and the saving did not need the warning anyway.

Why it matters. The lesson is worth more than the experiment and applies far beyond our small models. When someone shows you a technique for spotting the right moment to act during training, the natural question is whether the signal is real. Ours is real, and we have checked it several times over. The question people forget is what the watching costs, and that has to be subtracted from the saving rather than mentioned afterwards. An instrument has to pay for itself.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 24 training runs, no GPU, no cost.

Program v2 Bucket J, item J7. Decisive computation: analysis/spend_the_lead.py. Output: analysis/spend_the_lead.json. Reproduce with python analysis/spend_the_lead.py, or re-decide the target and budget from the saved series with --reuse.

The question

Everything in this programme measures. Nothing had yet tried to use any of it to save anything, which is awkward for a programme whose stated subject is compute efficiency.

**Switching to a cheaper setting helped. Timing it did not..** We can read a model's internal state and tell it is about to improve. This is the first time we tried to profit from that. At the moment the warning fired we cut the amount of data used per training step by four, which makes each step cheaper, and gave every version the same total data budget so the comparison is fair. The bars show how much data each version needed to reach the accuracy target. Lower is better. The third bar made exactly the same cut, but at a moment chosen at random instead of when the warning fired. The cut is worth having: both switched versions reached the target on about 22% less data than ordinary training. But the second and third bars are the same height. Making the cut when our warning fired was no better than making it at a random moment, so the warning, which was the whole point, contributed nothing. The manoeuvre works and the information does not.

**The detector cost thirteen times what it saved.** To act on a warning you have to keep checking for it, and checking is not free. The green bar is how much training data the manoeuvre saved. The red bar is how much data was consumed by the detector that decided when to act. Both are measured in the same unit, which is the only reason they can be compared at all. The detector cost about thirteen times the saving it produced, and the saving did not need it: cutting at a random moment saved the same amount for free. The lesson generalises past our small models. It is easy to check whether a monitoring signal is real and forget to subtract what it costs to compute. Ours is real. It still lost money by an order of magnitude.

F2's decode probe is the programme's one positive instrument. Its interpretation is specific: when the probe crosses, the answer is already linearly present in the hidden state and the head has not yet learned to read it. If that is what is happening, the remaining work is cheap, and from the crossing onward a cheaper regime should suffice.

So spend it. At the probe's crossing, cut the batch size by 4x (64 to 16) and carry on. Cost per step is linear in batch, so this needs no FLOP accounting and no wall-clock timing, which would be unmeasurable under a CPU throttle anyway.

Kill test, fixed in advance: on a compute-matched comparison there is no saving.

Design, and the fourth point is the one that decides it

The budget is compute, not steps. A programme standing control. Every arm gets 57,600 samples. An arm on a smaller batch gets more steps for the same samples, and that is the entire point of the manoeuvre: the control takes 900 steps, the switched arms take 3,399.
A random-time-matched arm switches at a random step drawn from the observed distribution of probe crossings. Without it, "cutting the batch at the probe's crossing helped" cannot be told apart from "cutting the batch helped". Also a programme standing control.
The probe is fitted online, as a real system would have to, rather than read off a finished run. The switch has to be decidable at the time it is made.
The probe's own cost is counted. This is the trap the item turns on. Detecting the crossing means fitting a probe repeatedly, and that is not free. A saving smaller than the cost of the instrument that found it is not a saving.

Result: the kill test fires, twice, for two different reasons

Eight seeds per arm, width 48, learning rate 0.005, matched budget of 57,600 samples. The probe crossed in all eight seeds, at step 65 or 70.

Final accuracy at a matched budget: the primary endpoint

Arm	Final accuracy	Steps taken
control (no switch)	`0.9917` `[0.9899, 0.9935]`	`900`
probe-timed switch	`0.9890` `[0.9861, 0.9920]`	`3,399`
random-timed switch	`0.9892` `[0.9859, 0.9926]`	`3,399`

No saving. The probe-timed arm finishes -0.0026 below the control and -0.0002 from the random-timed arm. Every interval overlaps every other.

Samples to reach 90% accuracy: the secondary, and it is more interesting

Arm	Samples to target
control	`11,080` `[10,380, 11,780]`
probe-timed switch	`8,620` `[7,976, 9,264]`
random-timed switch	`8,690` `[8,148, 9,232]`

Two things here, and they point in opposite directions.

Cutting the batch genuinely works. Both switched arms reach the target on 22% fewer samples than the control, and their intervals do not overlap the control's. That is a real effect, not noise.

The probe's timing contributes nothing to it. Probe-timed 8,620 against random-timed 8,690, intervals almost exactly coincident. The manoeuvre works; knowing when to make it is worth zero. The whole value of the instrument was supposed to be the timing.

The part that was not in the plan: the instrument is not free

Detecting the crossing cost 382,720 scored rows of probe fitting per run, against 691,200 rows of training, an overhead of 55.4%.

Put in the same units as the benefit:

	Samples
saved by switching, against the control	`2,460`
spent detecting when to switch	`31,893`

The instrument cost about thirteen times the saving it enabled, and the saving did not require it, because switching at a random moment saved the same 22% for free.

Verdict

The kill test fires. The lead cannot be spent this way. At a matched sample budget there is no accuracy gain, and the sample-to-target gain is entirely attributable to the batch cut rather than to its timing.
This is the highest-value negative in the bucket, as predicted. The programme's one usable instrument is not usable for this, which is worth knowing before any more instruments are built.
The generalisable lesson is the overhead, not the null. It is easy to evaluate a monitoring or early-stopping rule on whether its signal is real, and to forget to subtract the cost of computing the signal. This signal is real: F2, G6 and I1 all hold. It still lost money by an order of magnitude. An instrument has to pay for itself, and that belongs in the endpoint, not the discussion.
A new standing rule follows: any efficiency claim in this programme must price the measurement that produced it, in the same units as the benefit.

What this does not say

It does not say the probe is worthless. It says this particular way of spending it loses. Two readings survive:

The cheaper regime may simply be the wrong thing to switch to. F3 already rules out freezing the recurrence (it never reaches full accuracy), which is why a batch cut was chosen, but the space of cheaper regimes is larger than these two.
The overhead is an artifact of how often the probe is fitted, not of probes in general. A cheaper detector with the same lead would change the arithmetic, though J5 says the lead itself degrades with width, so a cheaper detector would be racing a shrinking prize.

Limits

Eight seeds, one task, one width, one learning rate, one crossing level, one cheaper regime. The 4x batch cut and the 0.5 crossing threshold were both fixed in advance and neither was swept.
The target for samples-to-target is 0.9, chosen before the runs. The --reuse path exists so a different target can be re-decided from the saved series without retraining; the record quotes the preregistered one.
Accuracy is near ceiling (0.99 in every arm), which is the standing hazard on this task. The samples-to-target endpoint exists precisely because the final-accuracy endpoint has no headroom, and it is the endpoint that carries the interesting part of the result.
The overhead figure is specific to this probe configuration (2,048 rows, 80 fitting steps, refitted until the crossing). It is a measured cost of the instrument as this programme actually runs it, not a general claim about linear probes.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
compute-matched: Comparing two methods by giving them the same amount of computing power rather than the same number of steps. A method that takes twice as long per step should not get twice the compute for free.
freezing: Stopping one part of a model from changing while the rest carries on learning. A way of asking what that part contributes, by taking it away.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
linear probe: A deliberately simple readout attached to a model's internals to test what information is present in them. Kept simple on purpose: if something complicated can extract the answer, the complicated thing may be doing the work itself.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.