How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Model selection

Choosing a Base Model Matters Less Than You Think

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

"Which model should we use" is usually the first question in an AI project and one of the last that should be settled. It is also the question a buyer is least equipped to answer, which is why so much vendor conversation happens there.

On the task we have measured most thoroughly, the answer matters considerably less than the effort spent on it.

Download the full paper (full paper, PDF)

The measurement

Six open-weight model families, three random seeds each, all fine-tuned with an identical recipe on the same 510 real commercial contracts, all scored on the same 99 held-out ones.

	Untuned	Fine-tuned
Best model	71.9%	86.0%
Worst model	56.9%	81.5%
Spread	15.0 points	4.5 points

The model that started last gained the most, 24.6 points, and finished within 5 points of the model that started first.

What follows for choosing a model

The starting score is a poor predictor of the finishing score. A 4B model that began 9.7 points behind the 14B leader ended 1.5 points behind it. If you evaluate candidate models by trying them off the shelf, you are measuring something that training substantially erases.

An unimpressive demo is weak evidence. The model that looks worst in an untuned comparison may be the one that gains most. Ranking candidates untuned, then training only the winner, is a reasonable-looking procedure that answers the wrong question.

Parameter count did not order the results. A 20B model finished fourth of six, behind an 8B and a 4B. Size bought neither the best score nor the largest gain, and it did buy substantially more hardware to serve.

What should drive the choice instead

If capability differences compress, the remaining criteria are the ones that do not.

Licence. A permissive licence, MIT or Apache-2.0, means your deployment does not depend on anyone's permission or future pricing decision. A custom licence may be perfectly usable and still needs reading before deployment, not after.

Provenance. Who trained it and where. For regulated and government-adjacent work this is a procurement question rather than a technical one. Either way it should be on the model card, so it can be a decision rather than a discovery.

Serving cost and hardware. A 4B model that lands within 1.5 points of a 14B is a very different operational proposition. When the scores converge, this is where the real money is.

Consistency across runs. In our matrix one model's gain varied by 6.0 points across three seeds on identical data, while another varied by 1.1. Given similar means, prefer the predictable one.

The limit on this claim is substantial. Compression is what we measured on contract extraction. It is not a law of fine-tuning. On another task in the same matrix, the spread between models widened after training rather than narrowing. We are not publishing that task's figures because its corpus is licensed for research use only, which means you are being shown the result that supports the point and told without numbers about the one that does not.

We check this specifically because an earlier claim from this same matrix did not survive being checked, and had to be withdrawn. A pattern observed on one task is a hypothesis about the next one.

What to do with this

Do not spend weeks on the selection. Pick two or three credible open-weight candidates and move on.
Weight licence, serving cost and stability over benchmark position, because those are the differences training does not erase.
Spend the saved effort on the data, which is where the larger measured difference was.
Re-measure on your own task before committing. Your corpus is not this corpus.
Be sceptical of a large advertised lift, including ours. Compare final scores.

Contract results are derived from the Contract Understanding Atticus Dataset (CUAD) v1, by Hendrycks, Burns, Chen & Ball, "CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review" (2021), The Atticus Project, used under CC BY 4.0. Complaint results use public-domain consumer complaint narratives from the US Consumer Financial Protection Bureau.

The measurement

What follows for choosing a model

What should drive the choice instead

What to do with this

Want this measured on your data?