How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 37 of 39

The Fix That Fixed Nothing

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. An earlier experiment of ours failed for a specific reason: we wanted to watch a model learn something, and it turned out an untrained model could already do it, so there was no moment of learning to look at. That failure told us exactly what a working version would need, so this is our attempt to build one. We had a theory about what went wrong and a redesign meant to fix it.

What we found. The theory was wrong, and we found out for free. The very first measurement, which needs no training at all, showed our redesigned task scoring the same as the original on exactly the thing we had redesigned it to change. We then had one dial left for making the task harder, and the only setting that hid the answer from an untrained model also stopped the trained model from ever finding it. One run in five got anywhere.

Why it matters. The reason is the useful part. Our dial did not make the task deeper, it made the evidence fainter, and faint evidence is hard for a trained model too. So the requirement for the next attempt is sharper than we could have stated before: the thing being learned has to be easy to see but hard to compute, rather than simply hard to see. The original question, which is the biggest unresolved one in this project, is still open. We are also pleased about the order we did this in: the check that killed our theory cost nothing, because we ran it before spending any computer time on the idea it was meant to support.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 5 training runs plus a free initialisation sweep, no GPU, no cost.

Program v2 Bucket I, item I5. Decisive computation: analysis/composed_world.py. Output: analysis/composed_world.json. Reproduce with python analysis/composed_world.py.

The question

H7 tried to separate this programme's two perfectly-correlated variables, architecture and objective. It could not run, and it ended by stating exactly why: a randomly initialised GRU already read its probe target at R2 = 0.77, and the prediction loss converged smoothly, so there was neither a hidden target nor an abrupt event. That turned H7 into a design specification. I5 builds a task against it.

**The setting that hides the answer also stops the model learning it.** An earlier experiment of ours failed because the thing we wanted to watch a model learn was something an untrained model could already do. So we redesigned the task to hide it, and measured, before training anything, how much an untrained network could still read off. The horizontal axis is the one dial we had for making the task harder. The green band is where the answer is hidden well enough to be worth training on. Only the leftmost setting hides the answer, and at that setting the model never learns it either: one run in five got anywhere, and none beat a do-nothing baseline by a meaningful margin. The dial does not make the task deeper, it makes the evidence fainter, and faint evidence is hard for everyone. Note also the right-hand pair: at the setting our earlier attempt used, our redesign scores the same as the original did, so the change we thought would fix it changed nothing. We found that out for free, before training anything.

The diagnosis I5 started from: H7's world clamps the agent at the walls, so position is a linear accumulation of the actions the model is given as input, and a random recurrent network is a good leaky integrator (F3). Make the walls reflect and position becomes a folded accumulation, a triangle wave, which is nonlinear in the action history.

Kill test: a randomly initialised GRU's probe R2 on the new target exceeds 0.3, or the task shows no abrupt event.

The diagnosis was wrong

The free initialisation check runs first and costs no training:

Step size	Side `R2` at init	Position `R2` at init	Under the `0.3` ceiling?
`0.10`	`+0.261` `[+0.245, +0.277]`	`+0.401`	yes
`0.25`	`+0.606` `[+0.601, +0.611]`	`+0.778` `[+0.766, +0.790]`	no
`0.40`	`+0.666` `[+0.645, +0.687]`	`+0.795`	no

At H7's own step size of 0.25, reflecting walls give position R2 = 0.778 against H7's clamping 0.770. Essentially identical. The wall behaviour was not the binding constraint, and the hypothesis this pilot was built on is refuted by its own first, free measurement.

What actually moves the number is step size, and that is the finding.

And the two halves of the specification pull in opposite directions

Only the 0.10 cell clears the initialisation ceiling, so only it was trained:

	Result
runs reaching `R2 = 0.5` on the hidden side	1 of 5
runs beating "predict no change" by 25%	0 of 5
runs with a resolvable crossing at all	2 of 5
final loss against the trivial baseline	`0.00543` against `0.00590`, 8% better

The model does not learn the target either. The cell that hides the target from an untrained network also hides it from a trained one.

The reason is structural, and it is the useful output of this pilot. In this world family the free parameter controls signal strength, not compositional depth. The sensor is symmetric, so which side the agent is on is resolved only by the asymmetry that accumulates as it moves; a small step makes the target hard to read at initialisation precisely because it is weakly identified from the data at all. A large step makes it identifiable and therefore readable by the reservoir too.

Hard-to-read-at-init and learnable are in tension whenever difficulty is tuned by weakening the signal. H7's specification cannot be met that way, and this pilot is the demonstration.

Verdict

The kill test fires. No cell meets both halves of H7's specification. H7 remains unrunnable and the architecture-versus-objective confound remains open, still the largest one in the programme.
The reflecting-wall hypothesis is refuted, by a free measurement, before any compute was spent on the arm it was meant to enable. That ordering is the design point worth copying.
The requirement is now sharper than H7 left it. Difficulty has to come from compositional depth, a nonlinear function of a strongly identified state, and not from weakening the signal that identifies the state. A target that is hard because it is faint is not hard in the way this needs.
What that means concretely for the next attempt: keep the state fully identifiable from the observations, and make the probe target a nonlinear function of it: a parity, a comparison between two distant moments, a count modulo something. The state should be easy; the question asked about it should be hard.

Limits

One world family, one architecture, one width, one learning rate, 5 seeds, 3 step sizes.
The comparison against H7's 0.770 is close but not exact: H7 used 16-step trajectories and this uses 24. The two numbers are 0.770 and 0.778 at the same step size, which is well inside what that difference could produce, so "reflecting walls change nothing here" is the right reading and "they change nothing at all" is not established.
The 0.10 cell may be underpowered rather than impossible. 900 steps at one learning rate is not a serious attempt to learn a weakly identified target, and a longer run or a tuned rate might reach it. The tension it demonstrates would still hold, because the mechanism is about identifiability rather than about optimisation.
Sharpness is measured on the probe curve, which is noisy at low R2; two of five runs had no resolvable crossing, and that is a property of the measurement as much as of the runs.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
baseline: The thing you compare against. A result without one is not a result.
confound: A second explanation you did not control for. If bigger models both learn faster and score higher, then 'fast learners score higher' may be entirely about size and not about speed.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →