How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Getting started

Is your data ready for AI?

By Joshua Bolick, Awareness Software Group ~6 min read August 2026

It is the first question almost everyone asks, and it usually gets one of two useless answers. A vendor who wants the engagement says yes. A vendor who wants a bigger engagement says no, not yet, but we can fix that for you. Neither is a measurement.

Here is the version we can defend, including the thresholds, because a readiness score you cannot interrogate is just a sales opinion with a number attached.

Three things decide it

Readiness is not one property. Our scorecard splits it into three, weighted by how often each one is what actually blocks a project:

Volume, worth up to 40 points. How much you have.
Shape, worth up to 35 points. Whether what you have is the kind of thing a model can learn from directly.
Quality, worth up to 25 points. Empties, duplicates and the other things that quietly poison a training set.

Add them up and you land in one of three bands: early below 40, workable from 40 to 69, strong at 70 and above. Workable is not a consolation prize. Most first engagements start there.

Volume gates everything else

Volume is scored on a logarithmic curve that keeps rising until about 2,000 records. That matters more than it sounds: a linear score would treat a client with 100,000 documents the same as one with 2,000, and the range above a few thousand is exactly where an engagement gets valuable.

Volume also caps the total, and it is worth saying why, because a free diagnostic from a vendor is exactly the sort of thing that quietly scores in the vendor's favour. Variety and quality both start near their maximum. So on an early version of this scoring, a single well-formed file reached 60 out of 100 and reported back "workable", and 25 records reached "strong". It was measuring tidiness and calling it readiness, which is flattering and useless.

Volume is now a ceiling rather than just a contributor, so no amount of variety or cleanliness earns a band the underlying data cannot support. These thresholds are pinned by a test, so they cannot quietly drift back toward the generous version:

Fewer than 10 records: capped at 35, which is early, whatever else is true of it
Fewer than 50: capped at 55. Enough to shape an approach, not enough to train on
Fewer than 200: capped at 69, which is just short of strong

One clarification that saves a lot of confusion: those thresholds count source records, not training examples. A single contract usually yields several training examples, so 200 documents is not 200 examples. It is generally rather more.

Shape matters more than format

People tend to worry about file formats. PDF or Word, database or export. That is mostly a solved problem and mostly not what decides this.

What decides it is whether your data carries the thing you want the model to learn. Documents, free text, code and chat threads are directly trainable: they contain worked examples of the judgement you are trying to capture. Tables and logs usually are not, at least not yet. They are records of outcomes rather than demonstrations of reasoning, so they need shaping into examples first. That is real work, and it is the honest reason a data-foundation stage exists.

So the question is not "what format is it in". It is "is the expertise visible in the artefact". A folder of finished contracts shows a model how your firm drafts. A spreadsheet of contract metadata does not.

A real number, including the part that is unflattering

We ran our own scorecard end to end against 40 real commercial contracts as a rehearsal. It scored 55 out of 100, workable. Not strong, and it could not have been: 40 records sits under the 50-record threshold, so 55 was its ceiling. The data was clean and well shaped and it still could not score higher, which is the ceiling doing exactly its job.

The useful output was not the number. It was knowing which lever to pull. More contracts, not more cleaning.

What "messy" actually costs

Less than people fear. Duplicates, empty files and inconsistent naming cost points but rarely decide anything on their own. The things that do genuinely block a project are narrower: not enough records, or records that do not contain the judgement you want to capture.

One deliberate design choice on our side. When a file cannot be read, it is counted and reported with a reason (needs OCR, needs an extractor, or genuinely unreadable) rather than dropped silently. A quiet drop turns 500 documents into 300 without telling you, and every number downstream is then computed on data you did not know you had lost.

How to check yours without sending it anywhere

This is the part that usually stalls. Finding out whether your data is ready has traditionally meant giving your data to someone who wants to sell you something. For a law firm holding client contracts, or anyone under a confidentiality obligation, that is not a small hurdle. It is often the whole reason nothing has happened yet.

So the scorecard runs entirely on your machine. You download it, point it at a folder, and it writes an HTML report locally. It opens no network connections at all, and that is enforced by tests rather than promised in a sentence. We never receive your data, your file names, or your score. If you want to talk about the result afterwards, you send us the report. That is your decision, made after you have seen what it says.

The result tells you your band, which of the three components is holding you back, which training approaches your data actually supports, and what would move the number. If the answer is "not yet", it says that, and it says what would change it.