How do you know a fine-tuned AI model is actually better?

You measure it. Hold out a portion of the client's real examples so the model never sees them during training, score the plain base model on those held-out questions first, fine-tune, then score again on the identical questions with identical scoring. The difference is the lift. Because the only thing that changed is the fine-tune, the improvement is attributable to it.

Does fine-tuning actually improve accuracy, or is the base model already good enough?

It depends on the task and, critically, on how much data you have. On a public benchmark of 510 lawyer-annotated commercial contracts, a base Qwen3-4B extracted short contract fields at 62.2% accuracy. Fine-tuned on 396 examples it reached 83.5-85.4%, a lift of 21.3 to 23.2 points across three runs. The same model trained on only 48 examples produced a lift inside the noise floor, so the amount of data was the deciding factor rather than the model's ceiling.

How much of our own data do we need to fine-tune a model?

Fewer examples than most people expect, but more than a handful. In our contract-extraction measurements, 48 training examples produced no measurable improvement while roughly 400 produced a 22-point gain. The honest answer for any specific case is that it depends on how varied and how consistent the examples are, which is what a paid readiness assessment measures on the client's own corpus.

Why can't you just use a public AI benchmark score?

Public benchmarks measure general ability on standardised tests, not your work. A model that scores well in the abstract can still be mediocre at reading your contracts or following your policies. Public benchmarks are also increasingly targeted during training, so a high score can reflect test preparation as much as capability. The only benchmark that answers your question is built from your own tasks.

Provenance and supply chain

What Is Actually In Your Fine-Tuned Model?

By Joshua Bolick, Awareness Software Group ~7 min read September 2026

Ask a commercial AI vendor what is inside the model you are paying for. In most cases there is no document to give you. Not a redacted one, not one under NDA: the artifact does not exist.

That is strange, because for ordinary software we settled this years ago. A software bill of materials lists the components in a build, where they came from and what licence each carries, and for anything sold into government it is routine. The model, which is the part actually making the decisions, has been the exception.

Our platform now emits one for every model it trains. This article is a real example, field by field, and an honest account of what it does and does not establish.

Download the full paper (PDF)

Why now

Two things moved in 2026, and one has been true for longer.

The standards caught up with AI. The two dominant bill-of-materials formats both grew extensions for machine learning: CycloneDX added component types for models and datasets with an embedded model card, and SPDX 3.0 added an AI Profile and a Dataset Profile. The vocabulary to describe a model in a machine-readable way now exists and is standardised.

The scope moved with them. CISA published its 2026 Minimum Elements for an SBOM on 29 July 2026, with NSA, FBI and fifteen international partners, replacing the 2021 elements. Ten new fields, four revised, and the scope explicitly extended to AI software.

One caution before anyone quotes that at you, including us. This is not a mandate. Agencies may request bill-of-materials data as part of risk-based review; there is no general requirement that every model ship one. Overstating that is exactly the kind of claim a technical buyer checks, and it is worth less than the document itself.

What the document actually says

The example below is generated from a real run: a contract-extraction fine-tune of a 4-billion-parameter open-weight model, on 60 examples with a 48/12 split. It is a smoke-scale run, not a claims-grade result, and its own document says so. It is here to show the artifact, not the model.

Where the base model came from

This is the question a procurement reviewer asks first, and the one an API vendor cannot answer:

"publisher": "Alibaba Cloud (Qwen team)",
"licenses": [{"license": {"id": "Apache-2.0"}}],
"properties": [
  {"name": "camp:model:originCountry", "value": "China"},
  {"name": "camp:model:released", "value": "2025-04"}
]

Publisher, country of origin, licence, release date. Whether a Chinese-origin base model is acceptable is a decision for the buyer and it varies enormously by sector. The point is that the buyer gets to make it, on a fact rather than an assumption.

The licence decision, carried rather than logged

Our pipeline gates on the base model's licence before training starts. That decision used to live in a log file nobody would ever read again. It now travels with the model:

{"name": "camp:governance:licenseDecision", "value": "allow"},
{"name": "camp:governance:licenseClass", "value": "permissive"},
{"name": "camp:governance:licenseReason",
 "value": "permissive license, cleared for commercial delivery"}

The derived model inherits the base licence, and the document records that chain rather than leaving your counsel to reconstruct it.

Numbers with the baseline they were measured against

A quoted accuracy figure with no baseline is the single most common way an AI result misleads without anyone lying. So every metric in the document carries the base model's score beside the tuned one. That is the same argument as our published account of six measurements that did not survive checking, applied to the supply chain.

The part that matters most

The limitations block is computed from the run, not written by a person deciding what to disclose:

Evaluation split contains 12 example(s), too few to detect
small differences reliably.

field accuracy: base 68.2% -> tuned 72.7% (+4.5% absolute)
  field 'agreement date': 87.5%  (base 62.5%)
  field 'document name':  71.4%  (base 85.7%)   <- worse than the base model
  field 'effective date': 57.1%  (base 57.1%)

Read the third line. On one of the three fields, the fine-tune made the model worse: 85.7% down to 71.4%. The headline number is a real gain and that regression is real too, and both ship in the same document, automatically, without anyone having to remember to mention it.

This is the whole reason the document is worth anything. A bill of materials that reported only the improvement would be a brochure with a schema. The disclosure has to be produced by the same process that produces the claim, or it depends on somebody choosing to be honest on a bad day.

It also tells you something practical: on that field, use the base model. You could not have known that from a headline accuracy figure.

The boundary, stated rather than implied

The document says this in both formats, in these words:

Base-model licensing was evaluated by the licence gate and its decision is recorded below. Rights in the customer-supplied training data are declared by the data owner and were not independently verified by the platform.

We document what we received. We do not audit whether you held the rights to the data you gave us, and we are not in a position to. A document that quietly implied otherwise would be worse than no document, because it would transfer a risk without anyone noticing.

What we are not claiming

Three things, stated plainly, because the argument this whole site makes is that you should check rather than take our word for it.

We do not claim format conformance. The documents target CycloneDX 1.6 and the SPDX 3.0 AI Profile and are self-consistent, but they have not been run through the official validators. "Targets" is the honest verb.
We do not claim completeness against the 2026 CISA minimum elements. Weight-file hashes and an author signature are not implemented yet. Both are on the list.
We do not claim it is required. See above. It is useful, and it may be requested. Those are different things.

Why this follows from owning the model

The case for a private model you own usually stops at the weights: the file is yours, it runs on your infrastructure, nobody can withdraw it or reprice it. True, and incomplete. Owning a file whose contents you cannot describe is a strange kind of ownership, and it is the version an auditor is least impressed by.

The document costs nothing to produce, because a pipeline already knows all of it: the model catalogue holds the provenance, the licence gate holds its own decision, the dataset builder holds the splits, the evaluator holds the harness. Nothing new is collected and no training run is needed. It describes a model that already exists.

That is why it is a fair test of any supplier. Ask for one. The answer, and how long it takes them to produce it, tells you whether the provenance was ever tracked or is being reconstructed for you now.