A Clean Benchmark Score Needs More Than a Data Promise

Picture the evidence packet that should sit beside a model’s benchmark score. It identifies the exact checkpoint that was tested, records when its training data closed, names the benchmark version, preserves hashes of the files and contains the evaluator’s run logs. If the packet has only a statement that the benchmark was “excluded from training,” the claim remains mostly unauditable.

That distinction matters because benchmark contamination, meaning exposure to test questions or close substitutes during training, does not require anyone to cheat during evaluation. A web crawl can collect a benchmark’s public repository, answer explanations in a paper, worked solutions on a tutoring site or copies embedded in another dataset. The evaluation harness can then operate exactly as documented while the score overstates how well the model handles unfamiliar work.

No single disclosure establishes that a benchmark was absent from every stage of training. The practical standard is narrower: model makers should show what they checked, which artifacts the check covered and what uncertainty remains.

Start with a claim that can be tested

“Not trained on benchmarks” is too broad. A useful claim names one checkpoint and one benchmark release, then covers pretraining, supervised fine-tuning and other post-training datasets that can change the model’s answers. Retrieval systems must also be separated from the model test, since a model with search access may fetch public solutions during the run even if its weights never encountered them.

The evidence packet should bind those moving parts together. A cryptographic hash, a short fingerprint that changes when a file changes, can identify the checkpoint, benchmark archive and scoring code without publishing their contents. Timestamps should show when training closed and when the test became available. Run logs should preserve prompts, system instructions, sampling settings, tool access, retries and raw outputs rather than only the final score.

This does not prove clean training data. It prevents a different problem: a provider quietly testing another checkpoint, changing the prompt or substituting a revised test while citing evidence collected for an earlier setup. The packet turns a general assurance into a claim that another party can challenge.

Public policy is beginning to require more training-data disclosure, but the legal requirement should not be confused with benchmark-level proof. The EU AI Act requires providers of general-purpose AI models to “draw up and make publicly available a sufficiently detailed summary about the content used for training,” using a template supplied by the AI Office. That is a disclosure obligation. It does not, by itself, require a public item-level ledger showing that every benchmark question and paraphrase was excluded.

Provenance records show what the pipeline knew

Data provenance records where a training item came from, how it was transformed and which dataset snapshot received it. In a mature pipeline, the model maker can search source URLs, document identifiers, text hashes and processing logs for the benchmark, its answer key and known mirrors. The team can also record a deny-list applied before training and retain the rejected matches.

This is stronger than a policy statement because an auditor can rerun the query against the recorded corpus. It is strongest when the provider retained normalized text, since exact file hashes stop matching after routine changes such as removing markup, altering whitespace or extracting text from a PDF. Near-duplicate search can catch lightly edited copies, though it introduces judgment calls about how similar two items must be before exclusion.

The limit is the boundary of the records. A manifest cannot describe data that a supplier omitted from its manifest, and a URL-level log may miss benchmark material copied into an unrelated page. Provenance also struggles with semantic leakage: a training example may teach the same puzzle and solution under different names without sharing enough text to trigger a match.

OpenAI’s GPT-4 technical report, for example, described checking benchmark material against training data and assessing contamination through matching procedures. Such disclosure is useful because it exposes the test method. It still cannot establish universal absence, particularly where the provider does not release the corpus or enough records for an outside party to reproduce the search.

Return to the evidence packet. A provenance report belongs there, but its scope line matters more than its page count: which corpus snapshots were searched, which transformations were covered, whether answer explanations were included and whether outside auditors could inspect the underlying records.

Canary strings detect some ingestion, not all learning

A canary string is distinctive text placed in a dataset so later searches can reveal that a copy was ingested. BIG-bench, a public collection of evaluation tasks, includes a canary marker intended to warn training-data builders that benchmark material should not enter a corpus. A provider that retains searchable training text can look for the marker and trace any match back through its pipeline.

A positive match is meaningful. It shows that at least part of the marked artifact crossed the monitored boundary, even if the provider later removed it before the final training run. The surrounding records must establish what happened next.

A missing canary is weaker evidence. A scraper may collect individual questions without the marker, preprocessing may strip it, or another website may reproduce the task content alone. Testing whether the trained model can emit a canary is weaker still: failure to recover a string does not show that the model never saw it, because training may not preserve the exact sequence in a form that prompts can elicit.

Canaries work best as pipeline alarms. They do not certify model innocence, and public canaries can be filtered while unmarked copies remain.

A test written after the freeze changes the evidence

The cleanest temporal argument starts after the checkpoint is fixed. An evaluator records the model hash, keeps the model isolated from retrieval and then administers tasks authored after that freeze. The test items cannot have been in the earlier training corpus because they did not yet exist.

This approach costs more than downloading a public benchmark. Subject-matter experts must write tasks, check answer keys, pilot ambiguous items and maintain secure handling. The resulting score may also be harder to compare with published leaderboards because other models were tested on different questions.

Held back does not always mean clean. A private test assembled before training could have leaked through contractors, shared drives or previous evaluation runs. Repeatedly querying a private set can also turn it into post-training data if developers tune prompts, fine-tune the model or select checkpoints based on the results. Each exposure spends some of the test’s value.

The evidence packet therefore needs an access log. It should record who received the test, when the model first answered it and whether results flowed back to the development team before the checkpoint was selected. A newly authored test administered once after the freeze offers a stronger claim than an old private benchmark with an undocumented history.

Third-party evaluation moves custody, not certainty

An independent evaluator can hold the test set, run the frozen model in a controlled environment and publish the protocol. That arrangement reduces the model maker’s opportunity to inspect questions, adjust prompts item by item or omit inconvenient runs. Raw output logs let the evaluator distinguish a scoring failure from a model failure.

Independence does not make a public benchmark uncontaminated. If the evaluator downloads a test that has circulated online for years, the model may already know it. Nor can an evaluator verify a closed provider’s training corpus without contractual access to records and systems. An API adds another problem: the provider controls the backend, so a model name may not guarantee that the same weights served every request unless the service supports version pinning or another form of attestation.

The strongest arrangement combines custody with novelty. The evaluator authors or commissions fresh tasks after the provider freezes the model, keeps those tasks outside the provider’s systems and disables browsing or retrieval during the run. Afterward, it publishes enough task design and scoring information to support scrutiny without immediately releasing every reusable item.

That setup costs money and delays publication. It is not necessary for every internal experiment. It is worth demanding when a benchmark score supports a safety claim, procurement decision or broad statement that one model has crossed a capability threshold.

For a buyer reviewing the packet, the decision rule is concrete. Treat provenance searches and canaries as evidence about the data pipeline. Treat a post-freeze, independently administered test as evidence about performance on unseen items. If the provider offers only an exclusion statement and a leaderboard score, describe the contamination status as unknown.

Questions people ask

Can a company prove that no benchmark data entered training?

Usually not as a universal claim. It can provide bounded evidence for a named checkpoint and benchmark version by preserving corpus records, searching for exact and near-duplicate material, documenting exclusions and showing that post-training datasets were checked as well.

Is a private benchmark automatically uncontaminated?

No. Private questions may have circulated among contractors, appeared in earlier evaluations or influenced checkpoint selection through repeated testing. A stronger test is newly authored after the model freeze, tightly access-controlled and used without feeding item-level results back into development.

Does finding a canary mean the model memorized the benchmark?

It proves that the marked artifact reached the part of the pipeline being searched, not that the final model memorized every question. Investigators still need records showing whether the item was filtered before training and whether unmarked copies or answer explanations remained elsewhere.

What should a buyer request with a benchmark score?

Request the model checkpoint identifier, benchmark version, training-data cutoff, contamination-search method, exclusions, test-access history and raw evaluation logs. For consequential claims, ask for an independently run post-freeze test; without that, the score may still be useful, but unfamiliar-task performance remains unproven.