Your AI Feature Changed Without a Release. Here’s How to Audit It

Consider a support assistant answering one recurring question: “Can I return opened headphones after 20 days?” The application retrieves the store’s return policy, sends relevant passages to a hosted model, applies safety controls, and shows the answer with a citation.

The code may be identical on Monday and Friday while the answer changes from an eligibility decision to “contact support.” A model alias may now resolve to a different snapshot. The retrieval index may contain a revised policy or return different passages. A safety threshold may have moved in a provider console.

None of those changes requires a software release.

This support assistant is a useful audit target because its expected behavior is narrow. It should identify the applicable policy, avoid promising an exception, cite the policy it used, and hand off when the evidence is missing. The audit must explain which component altered that behavior, not merely confirm that the repository has no new commit.

Define the served configuration

Start with a rule that treats the application build as one field among several. A workable internal requirement is:

“For every user-visible AI response, retain the application build, requested and returned model identifiers, prompt revision, retrieval corpus revision, safety-policy revision, and enough sampled evidence to reconstruct the decision.”

This is a proposed change-control requirement, not a requirement imposed by a named regulation. Teams can enforce it at the AI gateway, the service that assembles prompts and calls providers, by refusing production traffic when required revision fields are absent. A dashboard that displays the same fields is useful, but it does not enforce anything.

The record for the opened-headphones request should include a request ID and timestamp; the application build; the model name requested by the application; the model identifier returned by the provider; and a provider fingerprint, when one exists. OpenAI’s model documentation describes snapshots as a way to hold model behavior more consistently than a moving alias, while its API exposes fields such as `model` and, for supported endpoints, `system_fingerprint`. A fingerprint can indicate that backend configuration changed. It does not explain the change or guarantee identical output.

Record the prompt template revision separately from user text. A content hash, a one-way digest used to detect whether content changed, is useful for comparison, but a hash cannot replay a prompt after the original has been deleted. If exact reconstruction matters, retain the rendered prompt under access controls or store it only for an approved sample of requests.

Give retrieval data an immutable revision

The support assistant does not answer from the model alone. It queries a retrieval index, a searchable representation of company documents, then places selected passages in the prompt. Replacing the documents behind an index name such as `support-production` can therefore change the answer without touching the model or application.

Do not log only that logical name. Publish each policy corpus under an immutable revision, then make `support-production` a pointer to the approved revision. For the headphones request, retain the corpus revision, namespace, retrieval query, selected record IDs, relevance scores, and hashes of the passages sent to the model. If policy permits, keep the passages themselves in a restricted audit store.

Pinecone’s documentation, for example, describes namespaces and record metadata that applications can use to separate and filter data. Those mechanisms do not create a business-level corpus revision automatically. The team loading the records must assign one, associate each record with it, and prevent in-place edits that erase the previous state.

This costs storage and operational work. It also gives the team a direct test: replay the same question against the prior corpus revision. If the old index retrieves a passage allowing opened-item returns while the new one retrieves a policy exclusion, the data change explains the behavior. If both produce the same passages, investigation moves downstream.

Move safety settings into controlled configuration

Safety controls are another production dependency. Some providers accept category thresholds in each request; others also expose defaults or account-level controls. Google’s Gemini safety-settings documentation, for instance, describes configurable blocking thresholds for supported harm categories and notes that safety behavior depends on the model and settings in use.

Pass explicit settings where the API permits rather than relying on provider defaults. Store them as a reviewed policy object such as `support-safety-2025-04`, with the category, threshold, exception handling, owner, and approval record. The date-like label here is a naming pattern, not a claim about a particular provider release.

For the headphones workflow, log whether input or output screening blocked the request, which policy revision made the decision, and what fallback the user received. Do not store a generic `blocked=true` field and expect an auditor to infer whether the model refused, a moderation endpoint intervened, or the application suppressed an otherwise completed answer.

A settings screen is evidence that someone can change a threshold. It is not adequate change control. Route production edits through an API or configuration repository that requires review, creates an immutable revision, and can restore the previous value. If the provider permits console changes that bypass this route, restrict that permission and alert on its use.

Treat provider notices as triggers, not proof

Assign an owner to each hosted dependency and subscribe that owner to provider changelogs, deprecation notices, status updates, and model lifecycle communications. Record the notice, the affected endpoint or model family, the assessment, and the resulting test run in the same change ticket.

A provider notice can tell the team to investigate. It cannot prove which configuration served a request, and it may not describe every backend adjustment that affects output. Polling an alias and recording the returned model identifier provides another detection path, although even a pinned identifier does not expose all infrastructure details inside a hosted service.

The enforced control is the gate that prevents an approved model, corpus, or safety revision from being replaced silently inside your configuration layer. Monitoring provider notices is detective control. It can shorten the time to discovery, but it cannot stop a provider-side change.

When a provider can alter behavior behind a stable identifier, decide in advance what evidence would pause the feature. For the support assistant, a failed policy-grounding invariant should disable direct eligibility answers and switch to retrieval-only excerpts or human handoff. That fallback is less fluent and may increase support volume, but it is auditable.

Test claims rather than wording

Exact-text regression tests are brittle because generated wording can vary even when the underlying decision remains acceptable. Build a sampled evaluation around the workflow’s claims and actions instead.

The opened-headphones case belongs in a fixed test set alongside policy questions with missing evidence, conflicting retrieved passages, multilingual phrasing, and instructions embedded in documents that try to override the system prompt. Run the set when a provider announces a relevant change, before promoting a model or index revision, and on a recurring schedule to detect unannounced drift.

Score whether the answer states the correct eligibility outcome, cites a passage from the served corpus revision, avoids inventing an exception, and hands off when no controlling policy was retrieved. Those are testable invariants. Tone and sentence similarity can be reviewed separately, but they should not hide a changed policy decision.

Run candidate configurations off the user path first. Shadow testing, which sends a copy of an input to a candidate configuration without showing its output, adds model and retrieval charges but need not add user-facing latency. Sending every production request to two models may be unnecessary and expensive; a risk-based sample plus the fixed regression set is often enough for a bounded support workflow.

Promotion should update one versioned manifest containing the model reference, prompt revision, corpus revision, retrieval parameters, and safety-policy revision. The gateway reads that manifest and writes its identifier into every request log. Rollback then means restoring a prior manifest pointer, rather than asking several teams to remember which settings were active.

Preserve evidence without copying everything

Full prompts and responses offer the best replay evidence, but they may contain customer messages, account details, or sensitive policy text. Logging all content indefinitely creates a second data store with its own access and deletion obligations.

Separate operational records from sampled evidence. Keep configuration identifiers, hashes, provider response metadata, token usage, retrieval record IDs, and decision codes for each request according to an approved retention schedule. Store full content for a smaller governed sample, encrypt it, restrict access, and link deletion workflows to the source customer record where required.

The tradeoff should be explicit. Hash-only logs can show that two requests used different passages, but they cannot reveal whether the difference mattered. Full-content logs support investigation, yet increase privacy and security exposure. For the headphones answer, retaining the cited policy passage may be more valuable than retaining unrelated customer text surrounding the question.

An audit should end with a reconstruction: this application build used this approved manifest; the manifest requested this model and corpus; these passages were retrieved; this safety revision allowed or blocked the output; and this fallback applied. If one link is missing, “no software release occurred” is not an explanation.

Questions people ask

Can a pinned model still change its output?

Yes. A pinned snapshot reduces movement associated with aliases, but generated output can remain nondeterministic, and a hosted provider may change backend infrastructure without exposing every detail. Record the returned model identifier and available fingerprint, then use behavioral regression tests rather than treating the model name as complete proof.

Should every

AI request store the full prompt and response?

Usually not without a defined need and retention policy. Store configuration identifiers and decision metadata for every request, then retain full prompts, retrieved passages, and outputs for an approved sample or higher-risk events. Hashes help detect changes, but they cannot reconstruct deleted content.

What should trigger an AI regression test?

Run tests before promoting a model, prompt, retrieval corpus, or safety-policy revision; after a relevant provider notice; and on a recurring schedule for unannounced drift. A changed returned-model identifier, provider fingerprint, retrieval distribution, or block rate should also trigger investigation under thresholds set from the system’s own baseline.

What is the safest fallback when the cause is unknown?

For a policy assistant, stop making the disputed eligibility decision. Show the retrieved policy passage, label that the assistant could not determine the outcome, and route the request to a person. The fallback sacrifices automation, but it prevents an unexplained configuration change from becoming a customer-facing promise.