How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research record

The Batch Law Holds at a Fixed Rate

By Joshua Bolick, Awareness Software Group ~12 min read September 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.

A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.

Part of a bigger question: What actually makes training cheaper? – One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.

In plain English

What we asked. A recent paper found a simple formula for when large language models learn a basic copying skill: feed them bigger batches of examples at a time, and they get there in fewer steps, by a predictable amount. They kept the learning speed the same throughout. We tested the formula on our much smaller model.

What we found. With the learning speed fixed, our model followed the formula closely, despite being a completely different kind of model on a made-up task. But when we raised the learning speed along with the batch, as practitioners usually do, the numbers changed by more than the difference between the two kinds of model.

Why it matters. In practice: a rule about batch size is really a rule about batch size and learning speed together. If you change one, check the other.

The rest of this page is the technical record: the design, every number, and the limits. It is written for a reviewer, and you do not need it to have understood the result above.

New here? How to read a research record

Start at the verdict. Every record states, before the experiment was run, what result would have made us abandon the idea. That is the "kill test". Then it says whether the test fired. Nothing gets reinterpreted after the fact.
Numbers in square brackets are uncertainty. 23.4 [18.1, 28.7] means our best estimate is 23.4 and the true value is probably somewhere in that range. If a range includes zero, we cannot claim an effect.
Results that rule an idea out are kept. Roughly half of what is published here says an approach did not work, including plenty of our own. Those pages are the output, not a shortfall: knowing which direction is a dead end is what lets the next experiment go somewhere better, and most of what we now rely on came out of understanding why something else fell short. Work that only publishes what worked is not measuring anything.
Read the Limits section. Every record ends with what it does not show. It is the most honest part of any experiment and usually the shortest.
Pro tip: the figures near the top are designed to carry the result on their own. If you read nothing else, read the caption under each one, which says what it shows and what to take from it.

EXPLORATORY. Not a preregistered study. Local CPU, 72 training runs (80 cells, the batch-64 cell shared between arms). The design, anchor and kill test were committed (48d114b) before any run.

Program v2 Bucket T, item T8. Decisive computation: analysis/batch_law.py. Output: analysis/batch_law.json. Reproduce with python analysis/batch_law.py (about half an hour on a throttled laptop CPU); --reuse re-derives every endpoint from the saved series.

The question

arXiv 2511.16893 (Aoyama, Wilcox and Schneider) fits the step at which induction heads form in transformer language models as e^13.26 B^-0.37 C^-0.62 for batch size B and context size C, independent of model size from 50M to 7B parameters. Its methods do not vary the learning rate. This programme found that the learning rate decides when a transition arrives (O15-O19) and how strongly skills compete (T7). T8 asks whether the batch exponent travels to a tiny recurrent network, and whether it survives changing the rate with the batch.

**A published batch-size law fits our tiny model, until the learning rate moves.** A small model was trained with batches of 16 to 256 examples. Solid: the learning rate stays the same. Dashed colour: the rate is raised as the batch grows, by a standard rule. Grey: the curve a recent paper on large language models predicts. With the rate held fixed, our model follows the paper's curve closely: doubling the batch saves about a fifth of the steps. Raise the rate with the batch and it saves about a third. The published law describes one rate setting, not batch size alone.

**How strongly steps fall with batch size depends on the learning-rate rule.** One number summarising each curve in the other chart: how fast the steps needed fall as the batch doubles. The paper's value is for large language models trained at a fixed learning rate. *At a fixed rate our value sits close to the paper's. Changing only how the rate follows the batch moves it far more than the gap between a tiny recurrent model and a large language model.*

J8's own training loop (width-48 GRU, delayed copy, lag 4) and eight receivers, 800 steps, batch 16 to 256. Part A holds the rate at O16's tuned 0.006, as the paper did. Part B scales it by the square-root rule for adaptive optimisers, 0.006 * sqrt(B / 64). Endpoint: the first step at which held-out accuracy reaches 0.5, a level fixed by the task (chance is 1/32), interpolated between evaluations every 5 steps. Exponent: each receiver's log-log slope of that step against batch; interval over receivers.

Kill test, fixed before execution: Part A's exponent interval lies entirely outside [-0.55, -0.20]. Anchor, in code: batch 64 at 0.006 reproduces O16's transitions on all eight receivers. It holds.

Result: the kill test does not fire

Batch	Step to `0.5`, rate fixed at `0.006`	Step to `0.5`, rate `0.006 * sqrt(B/64)`
`16`	`127.5` `[119.8, 135.2]`	`182.1` `[175.1, 189.2]`
`32`	`94.7` `[89.1, 100.2]`	`113.3` `[107.9, 118.6]`
`64`	`75.3` `[71.8, 78.7]`	`75.3` (same runs)
`128`	`61.6` `[59.3, 63.8]`	`51.9` `[49.4, 54.5]`
`256`	`52.9` `[51.3, 54.6]`	`37.3` `[35.3, 39.2]`
Exponent	`-0.315` `[-0.329, -0.301]`	`-0.571` `[-0.585, -0.556]`

Every receiver reached the level at every batch, in both arms.

At a fixed rate, the paper's exponent describes this substrate: -0.315 against its -0.37, on a 17K-parameter recurrent network on a synthetic task rather than a transformer on text. Doubling the batch cuts the steps to the transition by about a fifth, not by half, so the number of examples to the transition grows with batch (B^0.685).

Scaling the rate with the batch moves the exponent by -0.256 [-0.280, -0.231] (paired, per receiver), to -0.571. Under the square-root rule a doubled batch cuts the steps by about a third, and examples grow only as B^0.43. The exponent is a property of the batch and the rate rule together, and a law fitted at one fixed rate is a statement about that rule as much as about batch size.

A shape the single exponent hides

From the table's means, the fixed-rate curve bends: each doubling of the batch saves less than the one before (local exponents -0.43, -0.33, -0.29, -0.22 from batch 16 upward). That is the shape of a critical batch size, past which a larger batch buys little. With the rate scaled the bend is gentler (-0.68 to -0.48). A single power law, the paper's or ours, is a summary across that range rather than a constant.

What stands

Kill test does not fire. At a fixed rate the batch exponent is -0.315 [-0.329, -0.301], inside the band around the paper's -0.37.
The rate rule changes the exponent by -0.256 [-0.280, -0.231]. The paper's law, fitted without varying the rate, should be read as holding at its rate.
Practical reading: to reach a transition in fewer steps with a bigger batch, raise the rate with it; without that, a doubled batch saves about a fifth of the steps and costs more examples.

Limits

One substrate, one task, one context length. The paper's context term was not tested.
One rate rule. A per-batch tuned rate could give yet another exponent; the square-root rule is the standard prescription, not the optimum.
The endpoint is a fixed accuracy level, not an induction-head measurement; the comparison is to the paper's functional form, not to its mechanism.

Terms on this page

Every piece of vocabulary this record uses, in plain language. Generated from the text above, so it cannot drift out of step with it.

accuracy: The fraction of answers a model gets right on questions it was not trained on.
batch size: How many examples the model looks at before updating itself once. Bigger batches give a steadier but more expensive update.
exponent: The number in a power law that says how strongly one quantity responds to another. A bigger exponent means a steeper response to the same doubling.
GRU: Gated Recurrent Unit. A compact design for processing sequences one item at a time, with internal switches controlling what it keeps in memory.
held-out: Data the model was never trained on, kept back specifically to test it. Scoring a model on data it has already seen measures memorisation, not learning.
induction: A model's ability to spot that a sequence is repeating and copy what came next last time. A standard early skill in language models.
kill test: A condition written down before running the experiment that says what result would make us abandon the idea. Fixing it in advance is what stops a disappointing result being reinterpreted as an encouraging one.
learning rate: How big a step training takes each time it updates the model. Too small and nothing happens; too big and it never settles.
parameters: The adjustable numbers inside a model. Training is the process of setting them. Model size is usually quoted as a count of these.
power law: A relationship where one quantity changes by a fixed percentage whenever another one doubles, rather than by a fixed amount. Most scaling results in AI are stated this way.
recurrent: A design that reads a sequence one item at a time, carrying memory forward. The main alternative is attention, which looks at everything at once.
slope: How steeply one quantity changes as another does. A slope of one between a warning and the event it predicts means the warning shifts exactly in step with the event.
transformer: The architecture behind most modern large language models. It uses attention to look at every part of the input at once, rather than reading in order.
width: How many internal numbers a model uses at each layer. The usual way we vary model size in these experiments.