A Rerun Won’t Prove What an AI Said. Preserve the First Run

A customer-support application generates a refusal that appears to contradict company policy. The reviewer refreshes the page, submits the same visible prompt, and gets a polite approval instead. At that point, the second response has not disproved the first. It has created another event.

That support refusal is the anchor for this procedure. The collection target is not merely the sentence on screen, but the full execution that produced it: layered prompts, model identifier, generation settings, retrieved policy documents, tool results, output metadata, screenshots, and timestamps. Some of those records may already be disappearing while the dispute is being discussed.

A workable internal requirement is: Preserve the original output and the available execution context in a tamper-evident record before rerunning, editing, or redacting the case. That is a proposed control, not a universal legal standard. Litigation holds, sector rules, contracts, privacy duties, and internal retention schedules may impose different requirements, so the evidence owner should copy the applicable language into the incident record rather than summarize it from memory.

Freeze the original run first

Start with the application record that generated the refusal. Disable automated deletion for that record if the system permits it, restrict routine editing, and export the raw response before anyone clicks regenerate. A raw response means the content returned by the model endpoint plus available metadata, rather than text copied from the rendered page.

Preserve the surrounding conversation in order. The visible user message may be only one layer. Many applications send a system prompt, which sets general behavior, and a developer prompt, which supplies application instructions, alongside conversation history, tool definitions, account attributes, and hidden formatting. Export the exact strings sent for the disputed request, including whitespace and template output.

Save the template source separately because it may have changed after the event.

Do not repair malformed text or remove sensitive material from the master copy. Restrict access to the original, then create a documented working copy for redaction. Otherwise, the team may preserve a cleaner artifact while deleting the evidence needed to explain why the support refusal occurred.

Record the model as the application saw it

Write down the provider, API endpoint, deployment or model identifier, account or project, region when exposed, and provider request ID. Capture whether the application called a hosted model, an internally deployed model, or a routing service that selected among models.

A model name is often an alias rather than a permanent description of the software that ran. Providers can update hosted systems behind stable names, while an internal gateway may reroute traffic after a capacity problem. If the provider exposes a dated snapshot, revision, deployment digest, or weights hash, preserve it. If it exposes none of those, record that limitation plainly.

Do not replace an unavailable revision with the current model name and call the run reproducible.

Save the generation settings sent with the request, including temperature, top-p sampling, maximum output length, stop sequences, seed, response format, safety configuration, and tool-choice controls when applicable. Temperature and top-p influence token selection, while a seed is an input intended to make sampling more repeatable under supported conditions. A seed does not guarantee identical output if infrastructure, model weights, prompt assembly, or service implementation changes.

Defaults matter too. If the application omitted a setting, write omitted rather than guessing the provider default. The difference can explain why a later test, conducted through a console with different defaults, does not match the disputed support refusal.

Preserve what retrieval added

The support application may use retrieval-augmented generation, a method that searches a document collection and inserts selected passages into the model’s context. Saving the final prompt is useful, but it may not show why those passages were selected or whether a document changed later.

Export the retrieval query, filters, collection or index identifier, ranked results, document IDs, chunk text, source locations, relevance scores when available, and the order in which passages entered the prompt. Preserve the underlying document bytes or a versioned repository reference, subject to access controls. A URL alone is weak evidence because the page can be edited while keeping the same address.

Also capture the application’s policy version and the user attributes that affected retrieval, such as product tier or jurisdiction, without expanding collection beyond what the investigation needs. If the disputed refusal cited an obsolete returns policy because an old chunk remained in the search index, a fresh rerun against the corrected index will hide the failure rather than reproduce it.

Some systems cannot snapshot an entire vector index, which stores numerical representations used for similarity search. In that case, preserve the returned chunks and their metadata, note that the broader index state is unavailable, and retain configuration records showing the embedding model, filters, and retrieval limit. The gap belongs in the evidence record.

Capture the screen, then capture better evidence

Take screenshots of the refusal in its original interface, including the conversation identifier, visible model label, browser address, error banners, citations, and local clock if practical. A short screen recording can preserve expandable citations or a tool trace that disappears after navigation.

Screenshots establish presentation, not full execution. They can omit hidden prompts, crop timestamps, flatten interactive content, and lose machine-readable identifiers. Preserve the page source or application export when useful, along with relevant server logs and a network archive if collection is authorized and it does not expose unrelated credentials. Never ask a reviewer to place tokens or passwords into an evidence bundle.

For each artifact, record the collector, collection time, source system, method, and original filename. Use Coordinated Universal Time for comparison across systems, retain the displayed local time and offset, and note whether clocks were synchronized. Save both the client submission time and server receipt or completion time when available. A disagreement between them is evidence about clock quality, not a reason to choose the more convenient timestamp.

Make the package tamper-evident

Place the collected files in a case directory with a manifest, which is an index describing every artifact. The manifest should connect the support refusal to its prompt export, raw API response, retrieval bundle, configuration, screenshots, and logs. It should also identify unavailable records, retention deadlines, and any redacted working copies.

Calculate a cryptographic hash for each file, a digital fingerprint that changes when file contents change, and store those values in the manifest. Hashes do not prove that a record was truthful when created. They help show that the collected file has not changed since hashing.

Use storage with restricted permissions, version history, or write-once retention where the risk warrants it. Log every transfer and transformation. If an investigator converts a browser archive to PDF, the archive remains the original artifact and the PDF becomes a derived copy with its own hash. This chain of custody costs storage and staff time, but rebuilding it after an escalation is usually impossible.

The collection also creates privacy risk. Prompts and retrieval results can contain customer data, employee notes, or licensed material. Limit the package to authorized reviewers, apply the relevant retention schedule, and separate preservation from broad internal sharing. Evidence integrity does not require unrestricted access.

Treat reruns as tests, not replacements

Only after preserving the support refusal should the team rerun the workflow. Give every rerun a new identifier and collect it under the same procedure. Start with the closest available configuration, then change one material variable at a time, such as the model snapshot, retrieved policy version, or sampling setting, so the comparison can support a causal investigation.

Repeated outputs can show whether the refusal is common under captured conditions. They cannot establish that the original output never occurred, and an identical answer does not prove an identical execution path. Hosted services may batch requests differently, safety systems may change, and retrieval ranking can shift when an index is updated.

The honest reproducibility claim is bounded: the package preserves enough information to inspect the original run and test similar runs under documented conditions. It does not promise identical text from a probabilistic system, which selects among possible next tokens, especially when the provider does not expose a fixed model revision or complete server-side trace.

Return to the support refusal before closing the case. The final record should say which input or configuration probably contributed, which evidence supports that finding, what remained inaccessible, and whether the result changed after a controlled correction. Keep the first refusal attached. The improved rerun belongs beside it, never over it.

Questions people ask

Is a screenshot enough to prove what the AI displayed?

A screenshot can document visible wording and interface context, but it rarely captures the full request, hidden instructions, model revision, retrieval results, or machine timestamps. Preserve it as presentation evidence alongside the raw response, conversation export, request identifiers, and relevant logs.

Can a seed make the disputed output reproducible?

A seed may improve repeatability when the provider supports deterministic sampling and the rest of the execution remains fixed. It cannot compensate for changed model weights, infrastructure, prompts, tools, retrieval results, or safety layers. Record the seed, but do not describe it as a guarantee.

What if the provider does not disclose the exact model version?

Record the model alias, endpoint, deployment, request ID, account, region, and collection time, then state that the underlying revision was unavailable. That limitation narrows the claim the organization can make. Application logs and provider support records may still connect the event to a service-side trace.

Should investigators rerun the prompt immediately?

Preserve the original run first because a refresh, regeneration, or document update can overwrite transient evidence. After collection, rerun under the closest available conditions and label every result as a separate test. The rerun can evaluate recurrence, but it cannot replace the disputed event.