How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 51 of 67

We Gave It The Answer Anyway

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. When one of these models suddenly gets good at its task, its internals rearrange, and we can measure how much. An earlier experiment of ours found that a model started off with another model's finished internal machinery barely rearranges at all -- it has the machinery already, so there is nothing to build. That suggested a sharper test. What if we do not hand over any machinery, and instead simply wire the correct answer into the model as a shortcut sitting alongside it?

What we found. It made no difference whatsoever. Models given a working shortcut rearranged themselves just as much as models that had to work the task out from nothing, and reached the point of suddenly getting good at almost exactly the same time. We checked carefully that the shortcut was genuinely in use rather than switched off and ignored: the model strengthens it during training, and on its own, with the network taken away, it gets every single answer right. A second version was given the shortcut but not told how to aim it, and aimed it correctly by itself in every run.

Why it matters. Putting the two experiments together tells us what the rearranging actually is. Giving a model finished machinery in its own parts quiets it; giving it a correct answer from outside does not. So the rearranging is the model fitting the task into its own components, which it does whether or not the answer is already available next to it. It also fits an uncomfortable pattern we found separately, where a rule with no parameters solves the same task perfectly and the trained model never quite catches it: the model does not take the easy route even when the easy route is wired directly into its own output. If you are evaluating a model, that is worth remembering -- what a system does internally is not well predicted by what shortcuts you have made available to it.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 30 training runs, no GPU, no cost.

Program v2 Bucket M, item M4. Decisive computation: analysis/built_in_algorithm.py. Output: analysis/built_in_algorithm.json. Reproduce with python analysis/built_in_algorithm.py, or re-derive every endpoint with --reuse.

The question

K7 showed that a model handed a learned mechanism, another run's recurrent weights, barely reorganises internally: it retains a fraction of a from-scratch run's expansion and transitions at step 20 instead of 84. This asks the same question with a designed mechanism.

**How much a model rearranges inside itself, when it is handed the answer.** When one of these models suddenly gets good, its internals rearrange, and we can measure how much. The first bar is an ordinary model that has to work the task out. The second was wired up with a shortcut that gives the correct answer directly. The third was given the shortcut but not told how to set it, and worked that out on its own. The last is a model that had already finished learning before we started, so it is our reference for how much rearranging happens with no learning at all. Being handed a working correct answer makes no difference: those models rearrange themselves just as much as one that had to work everything out. We checked the shortcut was genuinely being used -- it strengthens during training and gets every answer right on its own -- so this is not a case of the model ignoring something switched off. An earlier experiment found that handing a model finished internal machinery does quiet the rearranging. Together those say the rearranging is the model fitting the task into its own parts, which it does whether or not the answer is already sitting next to it.

The task is delayed copy and the algorithm is "emit the token k places back". So the architecture is given that operation outright, as an additive path beside the recurrent network, and the network is asked whether it still does the thing it does when it has to build the mechanism itself.

Kill test, fixed in advance: it still transitions, on K7's endpoint.

Arms

arm	what it gets
`plain`	an ordinary GRU. K7's `scratch`, and the anchor
`copy-gate`	the GRU plus a copy path at the correct offset, strength learned from a small start
`lag-mixture`	the GRU plus a copy path whose offset is a learned distribution over eight candidates, initialised uniform
`warm-all`	K7's no-event baseline, reproduced here so the retained share has a reference from these runs

The endpoint is K7's, imported rather than reimplemented: J4's expansion, the largest rise in residual representation energy over any window, in the moving and the frozen frame (D6), and the share of the from-scratch expansion each arm retains above the no-event baseline rather than above zero.

Result: nothing changes

arm	transitioned	final accuracy	transition step	expansion (moving)	expansion (frozen)
`plain`	6/6	`0.9875`	`84.0` `[81.7, 86.3]`	`+0.2474` `[+0.2342, +0.2605]`	`+0.4901` `[+0.4751, +0.5050]`
`copy-gate`	6/6	`0.9905`	`79.3` `[77.2, 81.5]`	`+0.2476` `[+0.2348, +0.2604]`	`+0.4890` `[+0.4742, +0.5039]`
`lag-mixture`	6/6	`0.9918`	`82.0` `[79.3, 84.7]`	`+0.2470` `[+0.2340, +0.2599]`	`+0.4893` `[+0.4749, +0.5038]`
`warm-all`	0/6	`0.9927`	never	`+0.0332` `[+0.0268, +0.0396]`	`+0.0511` `[+0.0391, +0.0630]`

Share of the from-scratch expansion retained, above the no-event baseline:

arm	moving frame	frozen frame
`copy-gate`	`+100.1%`	`+99.8%`
`lag-mixture`	`+99.8%`	`+99.8%`

The kill test fires, and there is nothing marginal about it. Handing the model a working copy path leaves the internal reorganisation at essentially exactly its from-scratch magnitude, in both frames, and moves the transition by about five steps. Both anchors reproduced: plain matches K7's scratch (+0.4896 frozen, transition 83.67) and warm-all matches K7's no-event baseline (+0.0583 frozen, never transitions).

The check that licenses the headline

A path the model never opens is a no-op dressed as an intervention. The first execution of this pilot reported the table above with no evidence that the copy path was used at all, which would have made the result meaningless: this programme has already published one control that removed 0.0% of what it was meant to perturb. So each built-in arm now reports three things at the end of training:

	`copy-gate`	`lag-mixture`
learned strength of the path (from `0.127`)	`0.371` `[0.346, 0.397]`	`0.668` `[0.573, 0.763]`
that path's own accuracy, network removed	`1.0000`	`1.0000`
selector mass on the correct offset (from `0.125`)	`1.0000`	`0.598` `[0.548, 0.648]`

The path is used, it is strengthened by training, and on its own it is exactly correct. The mixture arm was not even told which offset to read and put 60% of its mass on the right one, with its argmax correct in every seed, so it found the algorithm as well as being given the primitive.

And the network built its own machinery anyway.

What this says

K7 and M4 together separate two things that sound the same:

Handing a model a learned solution in its own weights removes the reorganisation (K7: warm recurrence transitions at step 20 and retains a fraction).
Handing it a correct, working, external algorithm removes none of it (M4: +99.8% retained).

The reorganisation is therefore not the model acquiring the function. It is the recurrent network fitting the task in its own parameters, and it does that whether or not the answer is already available beside it. The GRU does not outsource.

Read with M1, where a zero-parameter cache solves this task outright and the GRU never catches it, the picture is consistent and slightly uncomfortable: the network is not using the cheapest route even when the cheapest route is wired directly into its own output.

Verdict

The kill test fires. Both built-in arms retain +99.8% of the from-scratch expansion.
The manipulation is real: the copy path is strengthened during training and scores 1.0000 alone; the mixture arm found the correct offset unaided.
Both anchors reproduced, plain against K7's scratch and warm-all against its no-event baseline.
M4 is a negative for the constructive demonstration it was designed to give, and a positive for what the reorganisation is: parameter fitting, not function acquisition.

Limits

The copy path is additive at a learned strength, and that strength stayed small (0.37 and 0.67). It is enough to be correct on its own but it is not obviously enough to dominate the GRU's logits, so "the model was handed the answer" is true of the path and not necessarily of the output. A version that forces the gate wide open, or that replaces the GRU's contribution rather than adding to it, is a stronger test and is not run here.
Six seeds, one task, one width, one learning rate. The effect size makes six enough; a smaller one would not be.
The expansion endpoint is a best-window rise, which is positive under noise by construction, which is exactly why K7's no-event arm is the reference rather than zero.
Representation capture is delegated to the inner GRU, so what is measured is whether the recurrent part reorganises. A measurement over the combined model would be a different quantity and is not what K7 measured.
This does not show the reorganisation is useful. It shows it happens regardless. Whether the model would be as accurate without it is F4's kind of question, not this one's.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
argmax: Short for 'the argument that maximises'. A model outputs a score for every possible next token; the argmax is simply whichever one scored highest. Accuracy is measured on the argmax, which is why accuracy is a thresholded quantity: only the winner counts, and by how much it won is discarded.
baseline: The thing you compare against. A result without one is not a result.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
logits: The raw scores a model produces for each possible answer before they are turned into probabilities.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.