How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Research question

Does reshaping a model's internals make training cheaper?

By Joshua Bolick, Awareness Software Group ~2 min read August 2026

Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.

The question the whole project started from. If you push a model to keep its internal representation compact while it trains, do you get a cheaper model?

Where this stands

No. The technique does exactly what it claims and the claim does not translate into an efficiency gain -- ordinary training reaches the same place, and the overhead is 23x the measured benefit.

It is the question a client is most likely to ask a version of, and it has a full accounting rather than an opinion. The corrections and the method have proved more reusable than any positive result would have been.

Closed · 1 published result bears on this question.

How these results fit together

This question has one record because it is the one the project started with, and the answer turned out to be short. The idea was that a model's internal workings might be squeezed into fewer dimensions during training, and that squeezing them might make training cheaper. The squeezing worked -- we could remove almost all of the targeted structure and the model kept performing. That is exactly what makes the answer negative: if the model does not care whether that structure is there, then arranging it is not doing any work. The honest competitor settled it. Plain ordinary fine-tuning, with none of the machinery, reached the same place for less, and the penalty we had built cost more than the steps it appeared to save. Everything else on this site exists because that answer sent us looking for what *does* matter, and the first thing we found is that the question had been about the wrong object: the coordinates a model's activity is written in are cheap to rearrange, and the machinery that produces the activity is not.

The results

Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.

Representation Geometry Is Not Load-Bearing

A whole programme of work, written up. We could remove 97% of a model's internal spread for under a tenth of a percentage point of accuracy. Internal coordinates are not where the difficulty lives.

Read the record

Back to all research questions

Where this stands

How these results fit together

The results

Representation Geometry Is Not Load-Bearing

Want this measured on your data?