How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 32 of 39

The Warning Was Not a Quirk of One Method

By Joshua Bolick, Awareness Software Group ~9 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Models on this task suddenly get much better at around step 270, and we found we could see it coming: reading the model's internal state tells you roughly 24 steps early. That is the one genuinely useful instrument this whole programme has produced. But every time we had measured it, we had trained the model the same way, using one popular method.

What we found. That mattered because we had already been caught out by exactly this. A different internal timing result of ours turned out to hold only for that one training method and to fall apart under a simpler one. So we re-ran the warning under two much simpler training methods. It survived both, in every single run, and under the simplest one it actually arrives earlier.

Why it matters. The check that makes this readable is the first bar on the chart. We re-ran the original setup as a control, in a freshly written program, and it reproduced our published number exactly. Without that we would only know that some measurement gave some answer. One thing we deliberately do not claim: we cannot say why the simpler method gives more warning, because the methods were matched on how fast they learn rather than on their settings, so more than one thing differs between them.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 15 training runs, no GPU, no cost.

Program v2 Bucket I, item I1. Decisive computation: analysis/probe_optimizer.py. Output: analysis/probe_optimizer.json. Reproduce with python analysis/probe_optimizer.py, or re-decide the crossing level from the saved series with --reuse.

The question

F2's decode probe is the only positive instrument this programme has. G6 established it is the better of the two leading indicators and put a number on how far it can be trusted. Every measurement of it has been taken under AdamW.

**The early warning is not a quirk of one training method.** We can read a model's internal state and tell that it is about to get much better at its task, roughly 24 steps before its answers actually improve. Every previous measurement of that used one particular training method. Since we had already caught a different internal timing claim being an artifact of exactly that method, we re-ran the measurement under two much simpler ones. The warning survives, and under plain gradient descent it arrives even earlier. All three bars are comfortably above zero and every single run showed the effect. The left-hand bar is the check that makes the other two readable: it reproduced our published figure exactly, so the measurement is the same one. What we cannot read from this is why plain descent gives more warning, because the training methods were matched on how fast they learn rather than on their settings.

D8 is why that matters. It found the expansion itself survives plain gradient descent at full magnitude, but that the geometry peak's timing does not: the peak sits +0.5 steps from the transition under AdamW and +243 under SGD. This programme has already caught one timing claim being an artifact of adaptive optimisation, and it never checked this one.

Kill test: under SGD the probe's lead is within noise of zero, or reverses.

Result: it survives, and it is larger

Optimizer	Rate	Probe crosses 50%	Model crosses 50%	Lead	Resolvable	Leading
`adamw` (control)	`0.005`	`61.0`	`84.5`	`+23.5` `[+20.5, +26.6]`	yes	5/5
`sgd`	`1.6`	`53.6`	`86.0`	`+32.4` `[+27.9, +36.9]`	yes	5/5
`sgd-momentum`	`0.2`	`56.0`	`81.1`	`+25.1` `[+21.3, +28.9]`	yes	5/5

The AdamW control reproduces F2's published +23.5 exactly, to the decimal, on five seeds of a separately written pilot. That is what licenses reading the other two rows.

The lead survives without any adaptive scaling, and under plain SGD it is 38% larger. Every run in every arm leads by more than the measurement resolution.

The checks that decide whether that can be believed

Optimizer	Probe matches head at convergence	Final probe / model	Shuffled-label control
`adamw`	5/5	`0.9926` / `0.9923`	`0.0317`
`sgd`	5/5	`0.9680` / `0.9665`	`0.0316`
`sgd-momentum`	5/5	`0.9700` / `0.9636`	`0.0316`

The instrument check is run per arm on purpose. F2's first version reported the opposite verdict because its probe was under-powered, and the standing rule adopted then was that a null from a probe is a statement about the probe until validated against a known-achievable target. "The probe works under AdamW" would not establish that it works on an SGD-trained hidden state, so each arm validates its own. All three pass in every run, and the shuffled-label control sits at chance (1/32 = 0.0313) throughout.

Learning rates are D8's, screened there to match steps-to-solve rather than to be equal. Comparing optimizers at one learning rate compares learning rates.

Verdict

The kill test fires and F2 survives. The decode probe's lead is not an artifact of adaptive optimisation.
Unlike D8's timing claim, F2 needs no scope banner. This is the direct contrast worth recording: on the same task, at the same width, the geometry peak's timing is AdamW-specific while the probe's lead is not. The two timing claims in this programme behave differently, and before this ran there was no reason to expect that.
G6's reliability numbers are not AdamW-only in the sense that mattered. They were measured under AdamW and their generalisation is untested, but the phenomenon they measure is not optimizer-specific.
The lead being larger under SGD is unexplained and not something the design was built to answer. Noted as an observation, not a claim.

Limits

Five seeds per arm, one task, one width, one architecture, one rank.
The lead is measured at one crossing level (50% of each curve's range). F2 checked levels from 20% to 90% and found the lead positive at all of them; that sweep was not repeated here, and the saved series make it a --reuse away.
Three optimizers is not "optimisation in general". Adam without weight decay, RMSProp, and second-order methods are untested.
The SGD rate is 320x AdamW's, chosen by D8's screen. Matching steps-to-solve is the right control for this comparison but it means the arms differ in more than the update rule.
+32.4 against +23.5 is a difference between arms that this design cannot attribute. The intervals do not overlap, but the arms differ in learning rate as well as optimizer, which is exactly what the matching was for and exactly what makes the size of the lead uninterpretable here. Only its existence is being claimed.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
architecture: The blueprint of a model: what components it has and how they connect. Two models can be the same size and completely different architectures.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
hidden state: The model's working memory: the internal numbers it carries from one step of a sequence to the next.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
optimizer: The algorithm that decides how to change the model given the gradient. AdamW and SGD are two common choices and they behave differently.
probe: A small separate model trained to read information out of a bigger model's internals, used as a measuring instrument rather than as a product.
rank: How many independent directions a set of numbers really uses. A low-rank structure is one that looks high-dimensional but is actually simple underneath.
reliability: How often something works on an individual case, as opposed to how well it does on average. The two can differ a lot, and only one of them tells you what to expect next time.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
SGD: Stochastic Gradient Descent. The simplest training algorithm: take a step in the direction the gradient points, every time, with no adaptation.
weight decay: A common training setting that gently pulls a model's internal numbers toward zero, used to stop it over-fitting.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →