How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Measurement integrity

Six Results of Our Own That Were Wrong

By Joshua Bolick, Awareness Software Group ~7 min read August 2026

Any vendor can show you a number. The question worth asking is what would have had to happen for that number to be wrong, and whether anybody checked.

This is our answer, and it is deliberately specific. It describes six results we produced, believed for a while, and then found to be wrong. Every one was caught before it reached a client.

We publish this for a practical reason rather than a noble one. In this field the measurement is the product. If our numbers cannot survive us attacking them, they will not survive a technical buyer attacking them either, and it is much cheaper to find that out ourselves.

Download the full paper (full paper, PDF)

1 and 2. An untuned model that never answered

What we measured: a fine-tuned model scoring 72.7% against an untuned baseline scoring zero. On a separate task, a 45.5-point gain.

What was actually happening: some models reason step by step before answering. Asked to extract fields from a contract, the untuned model spent its entire output budget on reasoning and never produced an answer. Its own reasoning text, when read, frequently contained the right answers.

The comparison was never "the tuned model extracts better". It was "the tuned model answers at all", which is a much smaller claim. The second case was worse than uninformative: given a fair chance to answer, that baseline's score rose so far that the 45.5-point gain became a 9.1-point loss. We had the sign wrong, not just the magnitude.

What changed: the untuned model is now scored twice on every task, once allowed to deliberate and once not, and the better of the two becomes the baseline. We compare against the strongest available version of the thing we are trying to beat.

3. A hardware limit that was our own bug

What we measured: a particular model could not be evaluated on an 80 GB GPU. We wrote that into our hardware guidance.

What was actually happening: it was true that the evaluation failed. It was not true that the hardware was the reason. Our own code was holding two copies of the model in memory at once.

What changed: the bug is fixed, and the guidance was marked unknown rather than quietly reversed, until a fresh measurement settled it. A wrong claim corrected in the wrong direction is still a wrong claim.

This one is worth dwelling on because it is the least dramatic and the most common. The measurement was real. The failure was real. Only the explanation was wrong, and an explanation that fits the first observation is not a diagnosis.

4. A training technique that theory endorsed and measurement did not

Our own diagnostics said our training runs were stopping at the wrong point. The standard fix is well understood and we adopted it. Measured, it made accuracy worse by about two points and took three times as long. We reverted it the same day, and the measurement that killed it sits in the code so nobody re-enables it on the same reasoning.

The honest position afterwards is uncomfortable and worth stating: the original diagnosis still stands, the obvious fix does not work, and we do not currently know what the right one is.

5. A prompt that taught the model our file format

What we measured: an untuned model scoring 0% and a fine-tuned one scoring 88% on a document classification task.

What was actually happening: the untuned model was answering correctly. It was labelling its answer with one field name while our scoring looked for a different one, because our prompt never said which name to use. Fine-tuning taught the model our field name. That was the entire measured improvement.

How it was caught: the score was impossible, not merely surprising. With eight roughly balanced categories, guessing scores 12.5%. A zero meant something structural was broken.

A plausible but flattering number would have been far more dangerous than an absurd one. Had the untuned model scored 40% instead of 0%, the 88-point gain would have been a 48-point gain, nobody would have looked twice, and it would have gone in a deck.

6. An untuned model that answered with the instructions

What we measured: a fine-tuned model 53 points ahead of its baseline, which would have been the largest result in our entire matrix.

What was actually happening: the prompt asks for a reply in a specific format. Run in its reasoning mode, the untuned model replied with the literal placeholder word from the format example on 52 of 100 documents. It had copied the shape of the instruction rather than choosing an answer, and scored 35%. The correct baseline, the same model in its other mode, scored 79%. The real gain was 9 points, not 53.

How it was caught: automatically, by the check introduced after sections 1 and 2. No human noticed anything odd, and nobody had to.

What we do differently now

Check	The failure it catches
Score the untuned model both ways, keep the better	A baseline that never answers, making any comparison flattering
Read the raw model outputs, not only the score	A number that is arithmetically right and meaningless
Treat impossible scores as bugs, not results	0% or 100% almost always means broken plumbing
Prompts must name the field being scored	Measuring format compliance and calling it capability
Verify the fine-tune actually attached	Training that silently applied to nothing
Write predictions down before running	Hindsight reframing a disappointment as an expectation
Recompute every number from the raw files	Transcription errors. We have found several in our own tables
Three random seeds, never one	A result that is really seed luck

Why publish this at all

There is an obvious argument against it. A reader could reasonably ask why they should trust a vendor who has been wrong six times.

The answer is that every organisation doing this work has been wrong at least six times. The difference is only whether they measured carefully enough to notice, and whether they tell you. A vendor who cannot name a single result they had to withdraw has either not looked or is not saying.

Questions worth asking anyone in this space, including us

What did your untuned baseline actually output? Can I see it?
Was the baseline given the same chance to succeed as the tuned model?
How many runs is this number, and what was the spread between them?
Which parts of the task does your metric not measure?
What is a result you published and later withdrew?

The last one is the most informative, and the least often asked.