How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record 23 of 39

Being Choosy About Data Costs More Than It Saves

By Joshua Bolick, Awareness Software Group ~10 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

In plain English

What we asked. Training a model normally means feeding it whatever data comes next. An appealing alternative is to be selective: look at several candidate batches, work out which one teaches the most, and train only on that. The idea is that data covering ground the model has already learned is close to wasted effort.

What we found. It does not work here, and the reason is not the one we expected. We assumed selection would help and the problem would be affording it, since inspecting data costs time too. In fact selection did not help at all. We even tried a version that cheats, examining each candidate as thoroughly as actually training on it would, which no real system could afford. That version beat picking arbitrarily by half a step out of ninety-four.

Why it matters. This closes the idea at our scale rather than everywhere, and the distinction matters. The most likely explanation is our test data: every batch is drawn the same way from the same source, so there may be genuinely little to choose between them. On messy real-world data, where some examples really are more informative than others, the answer could differ. What we can say is that being selective is not automatically worthwhile, and that the usual argument for it, that you save the training you skip, assumes a benefit that has to be measured rather than presumed.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Negative results are kept. Roughly half of what is published here says an idea did not work, including several of our own. Those pages are not failures, they are the output. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 20 training runs, no GPU, no cost.

Program v2 tier 4, item B4. Decisive computation: analysis/gradient_novelty.py. Output: analysis/gradient_novelty.json. Reproduce with python analysis/gradient_novelty.py in about thirty minutes on a throttled laptop CPU.

The question

Score candidate batches by how much of their gradient is unexplained by recent gradients, and spend a full training step only on the novel ones. A batch pointing where training has already gone teaches little, so skipping it should buy back compute.

**Choosing which data to train on costs more than it saves.** Instead of training on whatever data arrives, look at four candidate batches, work out which one is least like what the model has recently seen, and train only on that. The green bars are how many training steps that saved. The red bars are how many training steps were burned deciding, since inspecting a batch is not free either. The three pairs use progressively cheaper ways of inspecting: the full learning signal, then just the output layer, then just the raw input. Every red bar dwarfs its green one, and it is not close. Being choosy about data does help a little, but the cheapest way we found to be choosy still costs several times what it returns. That is the whole question for this idea: it was never whether picking better data helps, it was whether picking can be made cheaper than the training it replaces. On this model, at this size, it cannot. The honest caveat is that the price of inspecting relative to the price of training is exactly the number that would change on a much larger model, so this closes the idea here rather than everywhere.

Kill test, from docs/research-program-v2.md: the whole difficulty is making scoring cheaper than the training it saves. That is the kill test, not the effect size.

Four arms, each training on exactly one of four candidates per step, so every arm takes the same number of training steps and differs only in which batch each step used:

Arm	How a candidate is scored	Cost per scoring pass
random	take the first candidate, the control	`0`
full	the whole gradient. The upper bound on benefit, and not a practical method: it computes exactly what it is trying to avoid computing	`0.92` steps
head	the output layer's gradient in closed form from the forward pass, `(softmax − onehot)ᵀ @ hidden`. One forward, one matmul, no backward	`0.25` steps
embed	novelty of the batch's mean token embedding. A lookup and nothing else	`0.01` steps

The first version's cheap arm was not cheap. It called loss.backward() and read only the head's gradient, so the backward still traversed the whole recurrence: 3.107 ms against the full gradient's 3.057 ms, identical. A rule needing a backward pass to decide whether to do a backward pass was never a candidate, and the smoke test caught the arm doing exactly that. The closed form above is what makes B4's kill test answerable at all.

The kill test never gets to apply

Arm	Steps to the transition	Overlaps the control?
random	`93.6` `[90.4, 96.8]`	,
full	`93.2` `[90.4, 96.0]`	yes
head	`92.4` `[89.7, 95.1]`	yes
embed	`95.6` `[93.5, 97.7]`	yes

Every arm is indistinguishable from picking the first batch that arrives. All five seeds solved in every arm, and every interval overlaps the control's. The best arm saves 1.2 steps out of 93.6, 1.3%, and the cheapest arm is 2.0 steps worse than random.

Most tellingly, the full-gradient arm saves +0.4 steps. That arm is the upper bound: it has perfect information about each candidate's gradient, at a cost no practical method could pay, and it does not beat choosing arbitrarily.

So the economics never arise. The question was whether scoring could be made cheaper than the training it saves, and there is no training being saved.

The bound closes it

Five seeds cannot prove a zero, so the honest statement is a bound rather than an absence. The control's interval is 6.5 steps wide, 6.9% of the transition, so a benefit smaller than about that would not have been detected.

That bound is still not enough:

Undetectable benefit is at most ~6.5 steps.
The cheapest possible scoring bill: the embedding proxy, 80x cheaper than a gradient, is 0.01 steps per candidate x 4 candidates x 93.6 steps = 3.5 steps.
The next cheapest, the closed-form head gradient, bills 23 steps.

So the embedding proxy is the only rule whose bill even fits inside the measurement error, and it was the worst-performing arm. A benefit large enough to pay for anything more informative than a lookup would have been visible, and was not.

Verdict

The kill test fires, and for a stronger reason than it anticipated. B4 was framed as an engineering problem: selection helps, but can selection be made cheap enough? On this task the premise fails first: gradient novelty does not select usefully at all, even given the whole gradient for free.

Consequence: gradient-novelty data selection is closed at this scale, and the follow-up is not a cheaper proxy. A cheaper proxy multiplies a benefit that is not there.

Limits

The benefit measurement is the transferable half; the cost measurement is not. Whether novelty selection picks usefully is a property of the method. The ratio of scoring cost to training cost is a property of this machine at this size: on a width-48 GRU on a CPU, fixed overheads are a large share of a 3.9 ms training step, which flatters cheap scoring. On a large model a forward pass is a smaller fraction of a training step, so the head and embedding proxies would look relatively better. That would matter if there were a benefit to scale up.
One task, one width, five seeds, AdamW, 420 steps.
Novelty is measured against a 16-gradient history with a fixed rule. A different definition, novelty against a longer horizon, or per-example rather than per-batch, is a different experiment. Per-example selection is the version most of the literature actually means, and this pilot does not test it.
Four candidates per step is one setting. More candidates give selection more to choose between and cost proportionally more; the ratio does not obviously improve.
The synthetic task's batches are drawn i.i.d. from one distribution, so there may be genuinely little to select between. This is the most likely reason the benefit is zero, and it is the strongest argument that the result is about the task rather than about the method. A task with heterogeneous or curriculum-structured data is the honest follow-up, and is not something this setup can provide.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

AdamW: A widely used training algorithm that adapts how big a step it takes for each individual parameter. The default in most modern AI training.
gradient: The direction and amount by which each of a model's internal numbers should change to do slightly better. Training is repeatedly following it.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
seed: The number that fixes all the randomness in a training run. Same seed, same run. Running several seeds is how you tell a real effect from a lucky one.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.

About this record. This page is the write-up of an experiment we ran ourselves. Every number on it comes from a run we recorded, and the script that produced each one is named in the text above. We publish these results and the data behind them here, including the ones that went against what we expected.

Why we do this. The same care shown above is what we apply to client work. We build custom AI models that you own and run yourself, using our own platform, and the reason we publish research like this is that a model is only worth what its evaluation is worth. If a supplier cannot show you the experiments that went against them, you cannot check the ones that went their way.

All research from this programme →