How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Consolidation Is a Measurement-Frame Effect

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

EXPLORATORY. Not a preregistered study. No new experiments, no training, no GPU, no cost. Reads only the committed results/ bundles, the captured activation tensors in analysis/representations.pt and the final output head in checkpoint.pt.

Program v2 tier 0, item D6. Decisive computation: analysis/consolidation_fate.py. Output: analysis/consolidation_fate.json. Reproduce with python analysis/consolidation_fate.py in about two minutes on CPU.

The question

The confirmed phenomenon is that rank-r residual energy rises about 50% around the accuracy transition and then falls back. The rise has been measured five ways. The fall, the thing this project has been calling consolidation, has never been decomposed. Three fates were posed in the backlog, each meaning something different:

**The same measurement rises or falls depending on how you measure it.** The usual way this project measures the 'spread' of a model's internal state re-picks its main directions at every measurement, like measuring a growing child against a ruler that quietly stretches. The fixed measurement picks the directions once and keeps them. Before the learning jump, both agree that spread increases (the fixed one sees nearly three times more of it). After the peak, they disagree completely: the usual measurement falls, the fixed one keeps rising. The increase before the jump is real: energy genuinely moves into new directions, and by more than previously reported. The apparent settling-down afterwards is not real: nothing shrinks. The model's main directions simply rotate to catch up with where the activity already is. A stopping rule built on that 'settling' would have been triggering on the ruler, not the child.

Decay, the expanded directions lose their energy. The expansion was transient activity.
Rotation into the read subspace, the expanded directions become the ones the head reads. The expansion is where the solution gets built.
Persistence as unused variance, the energy stays and the head keeps ignoring it.

The answer is none of the three. It is a fourth thing the backlog did not anticipate, and it changes what the word "consolidation" can be used for.

Method

For each of the 41 eligible task-only delayed-copy runs (all 41 qualified), take the final_norm capture at the step where residual energy peaks and freeze the tail subspace at the peak: the orthogonal complement of the top-r principal subspace, which is exactly the subspace whose energy share the confirmed rise measures. Then follow that fixed frame to the last capture.

Every quantity here has a nonzero chance value, a random (W - r)-dimensional frame already captures (W - r)/W = 0.8333 of any fixed direction, so two controls run alongside:

a random-subspace calibration, which came back at 0.8334 [0.8284, 0.8384], confirming the analytic chance level empirically;
a random-time-matched control, in the spirit of this program's standing rule: the same measurements on a tail frozen at a randomly chosen other capture step.

Only task-only runs are analysed; a penalised run drives residual energy through its own loss, which would manufacture the structure being looked for.

Result 1: the fall is a frame rotation, not a contraction

Between the peak capture and step 500, measured inside the frozen peak-tail frame:

Quantity	Peak → final	Runs favourable
Energy inside the frozen peak tail	`x2.1776` `[2.0181, 2.3371]`	41 / 41
Energy in its complement (the peak's retained subspace)	`x1.5096` `[1.4567, 1.5625]`	41 / 41
Total representation energy	`x1.8297` `[1.7328, 1.9266]`	41 / 41

The expanded directions do not decay. They gain energy after the peak, and they gain it faster than the subspace that was being retained. Consequently the frozen tail's share of total energy rises from 0.4808 at the peak to 0.5662 at the end, in 40 of 41 runs.

Meanwhile, measured in each capture's own moving basis, the way every previous analysis in this project measured it, residual share falls from 0.4808 to 0.4233, in 41 of 41 runs.

Both statements are true of the same runs. The reconciliation is that the top-r subspace is not a fixed set of directions: it is redefined at every capture as whichever r directions currently hold the most energy. After the peak, that redefinition captures more of the still-growing energy than it did before.

The post-peak fall in residual energy is the retained subspace rotating, not the expansion unwinding. Nothing contracts.

Result 2: the rise is real, and larger than reported

The same frozen-frame test applied to the rise gives the opposite verdict, which is what makes the first result credible rather than an artifact of the method:

Pre-transition capture → peak	Share change
In each capture's own moving basis	`+0.0998` `[+0.0847, +0.1149]`
In a frame frozen at the pre-transition capture	`+0.2783` `[+0.2489, +0.3077]`

Energy genuinely moves into those directions across the transition, by 2.8x more than the moving-basis measurement shows, because the moving basis chases the energy and so understates it. The confirmed rise is not weakened by this analysis; it is strengthened.

Result 3: the head under-reads the tail, but not because of the peak

Measure (chance = `0.8333`)	Value	Runs above chance
Peak tail's alignment with the output head	`0.6093` `[0.5935, 0.6251]`	0 / 41
Peak tail's overlap with the final retained subspace	`0.4164` `[0.3824, 0.4504]`	0 / 41
Random subspace, same measure (calibration)	`0.8334` `[0.8284, 0.8384]`	20 / 41

The output head systematically under-reads the expanded tail, and the final retained subspace sits mostly inside what was already retained at the peak. So fate 2, rotation into the read subspace, is excluded: the expanded directions do not become the ones the head reads.

But this is not a property of the peak. Against the random-time-matched control, paired within run:

Paired difference (peak frame − random-time frame)	Mean	95% interval	Material?
Head alignment	`+0.0060`	`[-0.0431, +0.0552]`	no
Overlap with final retained subspace	`+0.0759`	`[-0.0230, +0.1748]`	no

A tail frozen at an arbitrary capture behaves the same way. "The head reads the top of the spectrum and ignores the tail" is a standing fact about these runs, not something the transition creates.

Verdict against the kill test

The kill test was "no consistent post-peak structure across the 41 archived task-only runs." Structure is consistent: 41/41 on all three energy ratios, 40/41 on the frozen-share rise, 0/41 above chance on both subspace measures. D6 is not killed. But the structure found is not specific to the peak, and it reinterprets rather than explains the fall.

What this costs the program

A2 (consolidation onset as a stopping criterion) is undermined. Its signal is the post-peak fall, and that fall is a property of the measurement frame rather than an event in the representation. A2 should either be re-specified against a frame-fixed quantity or dropped. As written, it would be detecting the top-r subspace catching up with the energy.
Any use of the word "consolidation" now needs a frame. The term entered this project from the moving-basis curve. In a fixed frame nothing consolidates.
The confirmed rise is unaffected and understated. It should be re-quoted in fixed-frame terms in any future write-up.

Limits

Captures are every 50 steps, so the peak is located on a coarse grid: peak-minus-transition offsets span -10 to +110, against the +14.1 the 10-step metrics.csv grid gives. The frozen-frame conclusions do not depend on peak placement, a random-time frame shows the same pattern, but the peak itself is imprecise here. D0 (mid-run checkpoints) would fix this.
Only the final output head is archived. Head alignment is therefore measured with the end-of-training head at every frozen time. That is the right question for "did these directions become read?", but it cannot say what the head was reading at the peak. Also fixed by D0.
One architecture (tiny-gru), one task (synthetic-delayed-copy-v1), one optimizer (AdamW), one learning rate. D8 and E3 still scope all of it.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, Consolidation Is a Measurement-Frame Effect, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

All research from this programme →