Open Model Weights Still Leave Auditors in the Dark

A customer-support team wants to self-host an open-weight model that drafts replies to account-closure requests. The procurement ticket includes a downloadable checkpoint, a license, a model card, and the claim that the model is transparent because anyone can inspect it.

The checkpoint matters. Model weights, the numerical parameters adjusted during training, let the team run the model on its own infrastructure, test modifications, and preserve a known artifact rather than depend entirely on a vendor’s API. Yet an auditor reviewing a disputed reply six months later will need more than that file. The auditor must establish which data shaped the model, which prompt surrounded the customer’s message, what documents were retrieved, which software version processed the output, and whether a person approved the final response.

None of those facts can be recovered reliably from the weights.

A checkpoint answers one narrow question

Access to weights answers whether an organization can load and execute a particular model artifact. Depending on the license and architecture, it may also permit fine-tuning, local testing, or analysis of how parameter changes affect outputs. This is meaningful access, especially for teams that need local processing or protection from an API provider changing a model without notice.

It does not reveal the model’s training records. Two developers can publish similar checkpoints while disclosing very different amounts about source datasets, deduplication, filtering, human labeling, copyright controls, or the removal of personal information. A broad description such as “publicly available web data” does not let an assessor identify whether a disputed document appeared in training, under what permission, or how often near-duplicates occurred.

Weights also omit the evaluation harness, which is the code and procedure used to present test cases, score answers, and aggregate results. A benchmark table may lack prompt templates, sampling settings, excluded examples, failure counts, or information about whether developers tuned the model after seeing the test set. Without those details, an independent team can run a benchmark with the same checkpoint and still fail to reproduce the published claim.

The account-closure workflow adds another layer. Its system prompt, the hidden instruction that sets the model’s role and constraints, may tell the model to avoid admitting fault, direct certain customers to a retention queue, or cite only retrieved policy documents. Those instructions belong to the deployed application rather than the base checkpoint. A transparent model can sit inside an opaque product.

Regulation treats these disclosures as separate artifacts

The EU AI Act illustrates the distinction. For providers of general-purpose AI models, Article 53 requires them to “draw up and keep up-to-date the technical documentation of the model, including its training and testing process and the results of its evaluation.” It separately requires a public “sufficiently detailed summary about the content used for training.” Publishing weights does not satisfy those clauses by itself.

The law also creates a qualified exception for some models released under a free and open-source license. That exception depends on more than a download link: the model’s parameters, including its weights, information about the architecture, and usage information must be publicly available. Even then, the exception does not erase every obligation, and it does not apply in the same way to general-purpose models classified as posing systemic risk. Applicability also depends on the provider, model, release timing, and intended market, so teams should obtain their own compliance assessment rather than infer legal status from an “open” label.

These are binding legal requirements where the Act applies, subject to its implementation schedule and enforcement arrangements. By contrast, the US National Institute of Standards and Technology AI Risk Management Framework is voluntary unless an agency, customer, or contract makes part of it mandatory. A company may map controls to that framework without submitting to a government audit.

The enforcement distinction matters in procurement. “Aligned with,” “informed by,” and “based on” usually describe a vendor’s chosen method. A contract can turn a disclosure into an enforceable delivery obligation, while a marketing page cannot.

Test the transparency claim against six records

Return to the procurement ticket. Instead of asking whether the model is open, the reviewer should request six records and note who can inspect each one: the public, the buyer under confidentiality terms, an independent assessor, or only the provider.

1. The exact model artifact and license. Record the checkpoint identifier, file hash, architecture, tokenizer, inference code dependencies, and license terms. A cryptographic hash, a fingerprint derived from a file, lets the team verify later that it used the same artifact.

Check whether the license permits the intended commercial use, modification, and redistribution; the ability to download a file does not establish those rights.

2. Training-data evidence. Ask for source categories, collection periods, licensing or permission basis, filtering rules, deduplication methods, and documented exclusions. A full dataset may be unavailable because of privacy, security, contractual, or intellectual-property constraints.

The fallback should be evidence an assessor can test, such as dataset manifests, provenance records, removal procedures, and access for a qualified reviewer under controlled conditions. A provider saying that disclosure is impossible offers less auditability than one defining a restricted review path.

3. Evaluation materials. Require the test-set identity or construction method, prompt templates, scoring code, inference settings, repeated-run policy, and unaggregated failure categories. If examples must remain secret to limit benchmark contamination, an independent evaluator can run them in a controlled environment.

That costs staff time and money, but it carries more evidentiary weight than a leaderboard image whose underlying runs cannot be examined.

4. Application instructions and context. Preserve the system prompt, safety policies, retrieval settings, tool permissions, and document index used for each production version. Retrieval-augmented generation, which supplies selected external documents at response time, can change an answer without changing the weights.

For the account-closure assistant, the relevant evidence includes the policy document retrieved for that customer and the prompt template that framed it.

5. Deployment logs. Log the model hash or API model identifier, prompt-template version, retrieved-document references, tool calls, output, safety interventions, user edits, and approval event. Logs create their own privacy and security exposure, particularly when prompts contain account details, so retention periods and access controls need to be designed before collection begins.

Redaction reduces exposure but can also remove the evidence needed to reconstruct an incident.

6. Change and incident records. Document fine-tunes, quantization, prompt revisions, safety-filter changes, known failures, and rollback decisions. Quantization, which stores weights at lower numerical precision to reduce memory and computing needs, can alter output quality even when the underlying model name stays the same.

The fallback for an unsafe release should identify the previous checkpoint and application configuration, not merely promise that engineers can revert it.

A useful evidence table gives each record an owner, retention period, access level, update trigger, and verification method. Empty cells are findings. They show where the organization is relying on trust, whether the service uses open weights or a closed API.

Open and closed systems distribute the gaps differently

Self-hosting the checkpoint gives the support team control over application prompts, runtime configuration, and local logs. It also makes the team responsible for collecting those records, securing them, monitoring model changes, and preserving enough infrastructure to reproduce an old output. Weight access reduces dependence on a model vendor while increasing operational responsibility.

A closed service may withhold weights and training details yet supply stronger deployment evidence through version commitments, audit reports, contractual notification duties, administrative logs, and retention controls. Those materials do not reveal how the base model learned. They may still answer the narrower operational question of who sent what data, which service version responded, and what happened before a reply reached a customer.

The purchasing decision should follow the required audit question. If the concern is training-data provenance, neither local hosting nor API logs close the gap. If the concern is reconstructing a customer interaction, deployment records may matter more than parameter access. If independent modification and continued operation are essential, weights and license rights carry more weight.

For the account-closure system, approval should depend on a reproducible record for one test case: checkpoint hash, prompt version, retrieved policy, generated draft, reviewer edit, and final action. If the team cannot produce that chain during testing, the word “open” on the procurement ticket should not move the application closer to production.

Questions people ask

Does access to model weights reveal the training data?

No. Researchers may sometimes infer whether particular material influenced a model, but weights do not provide a dependable inventory of training sources, permissions, filtering decisions, or deleted records. Those claims require separate provenance documentation and, where appropriate, controlled access to supporting evidence.

Can an open-weight model be independently audited?

Yes, within limits. An assessor can test the downloaded artifact, inspect available code, and run chosen evaluations, but conclusions about training provenance or a production incident require records beyond the checkpoint. Audit scope should name the evidence available and the questions that remain unanswerable.

Is a closed AI service always less transparent?

No. A closed provider may offer detailed system documentation, evaluation access, version notices, and deployment logs while withholding parameters. That can support an operational audit, although it will not permit direct inspection or modification of the model weights and may leave training-data claims dependent on provider evidence.

What is the minimum record needed for a disputed AI output?

Preserve the model or service version, application prompt, relevant input, retrieved context, output, safety actions, and human approval or edit. Add timestamps and integrity controls so records cannot be changed unnoticed. Without that chain, the organization may know which model family it bought but not how the disputed result was produced.