How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 31 of 39

Learning It Again Is Not Like Learning It

By Joshua Bolick, Awareness Software Group ~13 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. When a model first learns this task, something measurable happens inside it: a particular internal quantity jumps at the moment its behaviour does. Everything we have measured about that jump has been at a model's very first learning event. So we did not know whether it marks learning in general, or only learning from nothing. That difference matters practically: if it marks learning in general, it could be used as a live signal that a model is currently reorganising itself, which is genuinely useful when training on data that keeps changing.

What we found. We trained a model until it had the task, then quietly changed the task underneath it and let it learn again. It did learn again, in all eight runs, and eight matched runs that were not given anything new produced nothing that could be confused for it. But the internal jump the second time was about a sixth the size of the first, even though the model's behaviour improved almost as much. It is small but it is real: it shows up in every run, and in both of the two ways we know how to measure it.

Why it matters. We are reporting this two ways on purpose. We had agreed in advance on a size below which we would not count an effect, and this falls below it, so by the rule we set ourselves the answer is no. But the measurement is clearly not zero either, and saying only 'no' would be misleading. What follows from both readings together is the same: most of whatever the model builds the first time is built once and then re-used, and the signal is too small the second time to be worth watching for. One caveat we want to be plain about: we changed the task, but not by much. A bigger change might well produce a bigger response, and we have not tested that.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

CONFIRMED on 24 seeds, 2026-08-24. J1 ranked this record's frozen-frame endpoint as the most fragile interval in the programme, clearing zero by 0.060 half-widths against I3's 0.042 before that one reversed. J2 re-ran it at 24 seeds. Both endpoints survive, and every number below is unchanged in direction and close in size. See the update.

EXPLORATORY. Not a preregistered study. Local CPU, 16 training runs plus screening, no GPU, no cost.

Program v2 Bucket E, item E7. Decisive computation: analysis/relearning_event.py. Output: analysis/relearning_event.json. Reproduce with python analysis/relearning_event.py in about thirty-five minutes on a throttled laptop CPU.

The question

Every expansion measurement in this programme has been taken at a model's first learning event, on a task it has never seen. That leaves the practically interesting reading untested. If the expansion accompanies any reorganisation, it is a candidate online detector of "this model is currently restructuring": a live question for training on nonstationary data, and a more useful role than the leading-indicator one that D3 closed. If it only ever appears the first time, it is a fact about learning from scratch.

**The internal signature is mostly about learning something the first time.** When a model first learns this task, a particular internal measure jumps. We wanted to know whether that marks learning in general or only learning from scratch, so we trained a model until it had the task, then changed the task underneath it and watched it learn again. The bars compare how much the measure moved in each case, always against a window placed elsewhere in the same run so that ordinary drift cannot count as a result. The shaded band is the size we agreed in advance was too small to matter. The second time is real but small: about a sixth of the first, and inside the band we had already called too small to count, even though the model's behaviour improved nearly as much as it did the first time. The right-hand bar is a model that was not given anything new to learn, and it sits flat, which is what tells us the small bars are not drift. So the signature is mostly a first-acquisition phenomenon, and it cannot be used as a live detector of a model reorganising itself.

E7 trains through the transition on delayed-copy at lag 4, then changes the lag to 8 mid-run and watches the model re-learn. Weights, optimizer state and the recurrence all carry over, so the second event is reorganisation of a trained model rather than acquisition by a fresh one.

Kill test: the second learning event shows no expansion. The signature is then specific to first acquisition from scratch.

Design, and the three standing controls it had to satisfy

A no-swap control trains for exactly as many steps on the original lag. Same seeds, same budget, same everything but what it is asked to learn second.
A random-time-matched window is measured in every arm, at a time drawn without reference to the event. Required by the programme's standing controls.
The primary endpoint is a fixed window, 50 steps either side of the event, not a trajectory maximum, "maximum after minus baseline before" is positive under noise by construction.
Both frames are reported. D6 showed the post-peak fall is the retained subspace rotating rather than the expansion unwinding, so an across-step geometry comparison must say which frame it is in: the moving basis each step recomputes, and a basis frozen at the swap.

The learning rate (0.002) was screened on two excluded seeds against a rule fixed in advance: the first task must be learned and plateaued with two window-widths to spare before the swap, so the "second event" cannot be the first one still finishing.

The swap produces a real second learning event

	Accuracy after swap	Final on new task	Final on old task	Second transition
swap (8 seeds)	`0.116`–`0.164`	`0.853`–`0.888`	`0.663`–`0.696`	`562`–`615`
no-swap (8 seeds)	`0.985`–`0.991`	`0.996`–`0.998`	`0.997`–`0.999`	none, 0/8

The swapped models fall to near-chance on the new lag, re-learn it over roughly 70–115 steps, and retain about two thirds of the old task. 8 of 8 produce a resolvable second event; 0 of 8 no-swap runs do, which is the control working: continuing to train produces nothing that could be mistaken for one.

Result: it recurs, and it is a sixth the size

Paired within run, event window minus random-time window:

Arm	Frame	Change	Clears `0`	Clears the `0.05` floor
swap	moving	`+0.0322` `[+0.0143, +0.0500]`	yes	no
swap	frozen at the swap	`+0.0158` `[+0.0009, +0.0307]`	yes	no
no-swap	moving, random time	`-0.0004` `[-0.0028, +0.0020]`	no	no
no-swap	frozen, random time	`+0.0002` `[-0.0020, +0.0025]`	no	no

For scale, the same endpoint at first acquisition on this task is +0.2010 [+0.1867, +0.2154] (H7's positive control, same width and rank) and +0.2239 in D8. Re-acquisition is roughly one sixth of that.

And the behavioural change is not six times smaller. First acquisition moves accuracy about 0.03 → 0.98; re-acquisition moves it 0.15 → 0.88. Comparable behavioural size, six times less geometric response.

Verdict: the letter and the intent diverge

By the letter, the kill test fires. The preregistered materiality floor is 0.05 and neither frame clears it, so there is no material expansion at re-acquisition.

By the intent, the honest answer is "much smaller, not absent." Both frames are positive with intervals clear of zero, paired against a random-time-matched window, in 8 of 8 runs, while the no-swap arm's identical endpoint sits flat to within ±0.003. Something real is happening and it is about a sixth the size.

Reporting only the kill test would overstate; reporting only the positive interval would understate. What follows from both readings together:

The signature is mostly a first-acquisition phenomenon. Whatever the transition builds (F3: it creates features rather than selecting them), most of it is built once and re-used rather than rebuilt.
It cannot serve as an online detector of reorganisation. That was E7's practical motivation, and a signal a sixth the size of the one the materiality floor was set against is not one to alarm on.
A2 is not rescued. D6 already undermined consolidation-as-a-stopping-criterion; nothing here restores it.

Limits

One kind of second event. Changing the lag keeps the task family, the vocabulary and the output space fixed. A swap to a genuinely different task: a different family, or a different vocabulary, is a larger reorganisation and could plausibly produce a larger response. This is the obvious follow-up and the result should not be read past it.
The old task is not fully forgotten (0.67 retained), so re-acquisition here is partly re-use. That is realistic for nonstationary training and it is also exactly why the response might be small: less needs rebuilding.
One width, one architecture, one optimizer, one rank, 8 seeds per arm.
The eval grid is 10 steps, so each 50-step window holds five points per side. The endpoint is a mean over those, not a peak.
The materiality floor was set against first-acquisition effects. A floor calibrated for re-acquisition would be a different and unpreregistered choice, which is why the raw intervals are reported alongside the pass/fail rather than instead of it.

Update, 2026-08-24: twenty-four seeds

J1's power audit ranked this record's frozen-frame paired endpoint as the most fragile interval in the programme: it cleared zero by 0.060 half-widths, nearer than I3's 0.042 before more seeds reversed that one. J2 re-ran the pilot at 24 seeds (the original eight kept and marked), same configuration, same controls.

Both endpoints survive

Endpoint	8 seeds	24 seeds	Half-width	Margin
swap, moving frame	`+0.0322` `[+0.0143, +0.0500]`	`+0.0329` `[+0.0255, +0.0403]`	`0.0179` → `0.0074`	`0.80` → `3.45`
swap, frozen frame	`+0.0158` `[+0.0009, +0.0307]`	`+0.0145` `[+0.0072, +0.0219]`	`0.0149` → `0.0073`	`0.06` → `0.99`

Half-widths shrank by 2.0x and 2.4x, close to the sqrt(3) = 1.73 a 3x sample predicts, and a little better. 24 of 24 swapped runs produce a second learning event; 0 of 24 no-swap controls do, against 8/8 and 0/8 before.

Nothing in the verdict changes. Both frames remain positive and clear of zero; neither clears the 0.05 materiality floor. The letter-versus-intent split stands, and the moving-frame interval no longer reaches the floor at all (+0.0403 against the earlier +0.0500), so the "immaterial" half is now more secure than it was, not less.

The contrast with I3 is the useful part

Both records were flagged as fragile. They failed and survived for different reasons, and the difference is diagnosable in advance:

	I3 (reversed)	E7 (survived)
point estimate on more seeds	moved a lot (`1.66x` → `1.02x`)	barely moved (`+0.0158` → `+0.0145`)
what was wrong	the discovery sample was biased, a pattern found by looking	nothing; the estimate was just imprecise

A fragile interval whose point estimate is stable is merely underpowered. A fragile interval found by noticing a pattern is a different and worse problem. I3's split was spotted in the data and then tested on the same data; E7's endpoint was preregistered in the pilot before any of it ran. The audit cannot tell those apart, but the follow-up can, and the ordering to prefer is obvious: re-run the ones that were found by looking first.

What is still open

J1 flagged 16 intervals across 5 files. This settles the most fragile one. modular_state (E8) contributes five more and is next in J2's queue.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
baseline: The thing you compare against. A result without one is not a result.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
subspace: A slice of a model's internal space: some directions kept, the rest ignored.
vocabulary: The set of distinct symbols a model can read and produce.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →