How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

Does the Effect Survive the Optimizer?

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

EXPLORATORY. Not a preregistered study. Local CPU, 48 training runs, no GPU, no cost.

Program v2 tier 2, item D8. Decisive computation: analysis/optimizer_transition.py. Output: analysis/optimizer_transition.json. Reproduce with python analysis/optimizer_transition.py in about eight minutes on a throttled laptop CPU.

The question

Every recorded result in this project trained with AdamW. The confirmed expansion, rank-8 residual energy rising about 50% around the accuracy transition, with 12 of 12 peaks landing within 50 steps of it, could therefore be partly an artifact of adaptive per-parameter scaling rather than a property of learning. D8 is the cheapest remaining threat to the confirmed result.

**Change the optimizer and the effect survives, but its timing does not.** Every result in this project was produced with one particular training algorithm (AdamW). If the effect were a quirk of that algorithm, it would not mean much. The blue bars show how large the internal reorganisation is under three different algorithms; the orange bars show how often that reorganisation happens at the same moment the model learns the task. The blue bars are the same height everywhere: the effect is real and not an artifact of the training algorithm. The orange bars are not: with plain gradient descent the reorganisation arrives hundreds of steps *late*. So the size of the phenomenon travels between algorithms and its timing does not, which matters, because several planned experiments wanted to use that timing as a clock.

Until program v2 tier 1 it could not be run at all: optimizer.name was read from the config and then ignored, so a run asking for SGD trained with AdamW and said nothing.

Kill test: the expansion is absent or an order of magnitude smaller under SGD.

Design

Three arms on the exact configuration the confirmation used: tiny-gru, width 48, delayed-copy, vocabulary 32, sequence 16, batch 64, lag 4, 900 steps, rank 8. Ten measurement seeds per arm.

Learning rates are matched on steps-to-solve, not copied. Plain SGD at AdamW's rate simply learns later, and a later transition is confounded with a different transition. Rates were screened on seed 991, deliberately absent from the ten measurement seeds:

Arm	Screened rates	Selected	Screen transition (reference: 90)
`adamw`	,	`0.005`	90
`sgd`	0.05 → 1.6	1.6	95
`sgd-momentum` (0.9)	0.005 → 0.2	0.2	90

All three arms solved the task in 10 of 10 seeds. SGD needed 320x AdamW's learning rate, which is what matching on steps-to-solve is for.

Result: the magnitude survives, the timing does not

Arm	Transition	Rise (moving basis)	Peak offset	Within 50 steps	Final accuracy
`adamw`	88	`+0.2239` `[+0.2084, +0.2394]`	`+0.5` `[-4.4, +5.4]`	100%	0.9913
`sgd`	92	`+0.2421` `[+0.2252, +0.2589]`	`+243.0` `[+124.3, +361.7]`	10%	0.9595
`sgd-momentum`	82	`+0.2273` `[+0.2103, +0.2442]`	`+170.0` `[+66.5, +273.5]`	0%	0.9586

The kill test does not fire. The expansion is 1.08x and 1.01x the AdamW reference under SGD and SGD+momentum: not absent, not an order of magnitude smaller, not even meaningfully different. All three clear the preregistered 0.05 materiality floor. The expansion is not an AdamW artifact.

But the confirmation's timing claim is. Under AdamW the energy peak is essentially simultaneous with the accuracy transition (+0.5 steps, 10 of 10 within 50), reproducing the published +2.1. Under SGD it lands 243 steps later, and under SGD+momentum 170 steps later, 1 of 10 and 0 of 10 within tolerance. The confirmation's headline "12 of 12 peaks within 50 steps" is a property of AdamW, not of learning.

Guarding against the max-minus-baseline bias

This project's standing rule is that "trajectory max minus early baseline" endpoints are positive under noise by construction and need a materiality floor plus independent timing or shape evidence. Under SGD the timing evidence fails, so the rise needs independent shape evidence, and a matched fixed window provides it, since it takes no maximum at all. Freezing the tail subspace at the last evaluation before the transition and measuring its share of energy exactly 100 steps later, the same elapsed time for every arm:

Arm	Frozen-frame rise, +100 steps
`adamw`	`+0.0650` `[+0.0496, +0.0804]`
`sgd`	`+0.1517` `[+0.1370, +0.1664]`
`sgd-momentum`	`+0.1860` `[+0.1750, +0.1970]`

Every arm expands, on an endpoint that is not a maximum and cannot be inflated by sampling more often. The SGD arms expand 2.3x and 2.9x more than AdamW over matched time.

A measurement error caught before publication. The first version of this pilot measured the frozen-frame change "from the transition to the peak". That is not comparable across arms here: the peak lands ~0 steps after the transition under AdamW and ~240 under SGD, so it contrasted a five-step window against a two-hundred-step one, and reported AdamW's frozen rise as +0.0165 against SGD's +0.1661. The fixed 100-step window replaces it. The moving-basis comparison was never affected, since all arms run the same number of evaluations over the same 900 steps.

What this changes

The program v2 constraint table needs amending. It records "the transition signal is coincident, not leading, the geometry peak sits +2.1 steps after the accuracy transition midpoint." That remains true under AdamW and is false under SGD, where the peak lags by 170–243 steps.

This does not open a door. A signal that lags by 243 steps is further from being a leading indicator, not closer. What it does is narrow the scope of every idea that treats the geometry peak as a clock: those are now explicitly AdamW-scoped until shown otherwise, and any controller keyed to the peak would need re-tuning per optimizer.

It also scopes D6. That result: the post-peak fall is a rotation of the retained subspace, not a contraction, was measured entirely on archived AdamW runs. Under SGD the peak sits hundreds of steps later, so the post-peak window is a different part of training, and D6's decomposition should not be assumed to carry over.

The good news is real and worth stating plainly: the phenomenon this program is built on is not an optimizer artifact. It appears at full magnitude under plain gradient descent with no adaptive scaling, no momentum, and no per-parameter state at all.

Limits

One architecture (tiny-gru), one task, one width (48), one step budget (900).
Three optimizers. Adam without decoupled decay, RMSProp, and Adagrad are untested, and the interesting axis may be "adaptive vs not" rather than "AdamW vs SGD".
Learning rates were matched on the transition step, which equalises when learning happens but not the shape of what follows, and the post-transition dynamics are exactly where the arms diverge. A matched-post-transition-rate design would be a fairer test of the timing claim specifically, and is the obvious follow-up.
The frozen-frame window of 100 steps is a choice, not a preregistration. It was fixed before the arms were compared, but a different window would give different numbers; the qualitative ordering (all arms positive, SGD larger) is what should be read from it.

The plain-language version. This page is the record: the design, the numbers and the limits. If you would rather have the story, Does the Effect Survive the Optimizer?, explained without the jargon, covers the same result for a reader who has never seen a training curve.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

All research from this programme →