Test the Replacement Before Your Model API Disappears

Take one customer-support endpoint. It receives a refund request, asks a model to summarize the ticket, calls an order lookup function when needed, and returns structured JSON containing a proposed reply and escalation flag. The endpoint works today under a model name stored in an environment variable.

That variable is the concrete migration problem.

A calendar reminder can prompt an engineer to replace the name before a provider retires it. It cannot show whether the replacement refuses more requests, chooses the wrong tool, takes longer at the busiest hour, or produces answers large enough to breach a token budget. Those differences become visible only when both models see comparable inputs.

OpenAI, Anthropic, and Google Cloud publish model lifecycle or deprecation documentation that identifies retiring models and suggested successors. Those pages establish the deadline and supported path. They do not certify behavioral equivalence for this refund endpoint, because the provider cannot test its prompts, tool definitions, customer language, or application-side validation.

Start with every name that can reach production

A model alias is a reusable name that may point to a provider-selected model version, while a snapshot identifies a more fixed version where the vendor offers that distinction. The inventory must capture both, because replacing an obvious model ID in application code does nothing if a gateway, deployment name, fallback route, or batch job still resolves to the retiring endpoint.

For the refund workflow, search source code, infrastructure configuration, environment variables, prompt-management systems, evaluation jobs, and scheduled scripts. Include development and disaster-recovery environments. A forgotten nightly job may stay quiet until the first run after retirement, at which point its failure appears unrelated to the migration.

Create one registry row for each call path: application owner, requested model name, provider, cloud region or deployment, purpose, expected response format, tools exposed, fallback model, traffic level, and retirement status. Record the resolved model ID from the API response when the provider returns one. If a hosted deployment hides the underlying version, record the deployment name and the version shown in its control plane.

The requested and resolved names belong in production logs from now on. For the support endpoint, the useful line also contains prompt-template version, response-schema version, tool-definition version, finish reason, input tokens, output tokens, and total request time. Without those fields, a later comparison can reveal that output changed but cannot separate a new model from a changed prompt or schema.

This first pass often changes the migration scope. One environment variable may feed four routes, and only one of them may expose a refund-related tool.

Send a copy, but suppress its effects

Shadow traffic is a duplicate request sent to a candidate system without showing its response to the user. Sample production requests after applying the same privacy controls used for the primary model, then send one copy to the current model and another to the replacement with identical prompts, tool definitions, parameters, and response schemas.

Do not let the shadow model execute write actions. A tool call, meaning a structured request from the model to an application function, can create a refund, send a message, or update a record if the surrounding system treats it as live. Route replacement-model calls to a recorder that validates the tool name and arguments but returns a fixture or a sanitized copy of the production tool result.

Read-only tools still need discipline. The shadow path can reuse a cached order lookup result when freshness is not under test, avoiding a second query and keeping both models on the same evidence. If tool selection itself is the subject, let both models request the lookup, record the decisions, then execute only the production model's request.

Sampling controls cost. Shadowing 10 percent of requests adds replacement-model inference to roughly one request in ten, although the dollar increase will differ if the successor has another token price or produces longer answers. Start with enough traffic to cover a full business cycle, including weekends if the endpoint runs then, rather than declaring success after a convenient daytime batch.

A replay set complements live shadowing. Build it from redacted historical requests that include the refund workflow's known hard cases: missing order details, hostile language, policy exceptions, malformed attachments, and attempts to manipulate the tool instructions. Replay gives every candidate the same inputs. Live traffic catches new patterns the saved set does not contain.

Compare decisions, not prose similarity

Text similarity is a weak gate for a support reply. Two answers can use different wording while reaching the same safe decision, and two nearly identical answers can disagree on whether to call the refund tool.

Start with refusals. Label an explicit refusal, a partial answer constrained by policy, an empty or truncated response, and a completed answer separately. Report the rate for each model and review the disagreements by category. A single overall percentage can hide a replacement that improves handling of abusive requests while refusing ordinary refund questions more often.

Next, compare tool behavior at two levels. The first is selection: did the model call the order lookup, return without a tool, or request an unavailable function? The second is arguments: did it supply the correct order identifier and required fields in the expected types? Exact argument-string matching is too brittle, so normalize JSON and compare typed values, while sending any financially meaningful disagreement to human review.

For structured output, record schema validity and application acceptance. A response can be valid JSON yet fail the endpoint because an enum changed, a required field is blank, or explanatory text appears outside the object. Run the production parser against both outputs. The replacement has not passed if engineers need to loosen validation merely to make the graph look green.

Latency needs the distribution, not one average. Track time to first token if the interface streams, total duration, and p95 latency, the point below which 95 percent of measured requests finish. Compare by input size and tool path because a fast no-tool answer can conceal a slower lookup route. The gate should come from the endpoint's existing service target, not a generic model benchmark.

Output length affects both cost and product behavior. Compare output-token distributions, truncation rates, and the share that reaches the application's configured maximum. A successor that writes longer replies may increase token spending and delay downstream text-to-speech or moderation, even when its per-token price is lower. Review the longest disagreements rather than relying on the median.

For the refund endpoint, one comparison record should now tie the same request to two outcomes: refusal label, selected tool, normalized arguments, parser result, first-token time, total time, and output tokens. Keep the production answer visible to operators and the shadow answer hidden.

Write cutover gates before seeing the result

Teams move goalposts when the replacement is already mandated. Set gates first.

The refusal gate should specify which categories may change and how disagreements are reviewed. The tool gate should require zero unapproved write actions in the shadow recorder, plus an explicit tolerance for lookup-selection and argument differences. Latency and output-length gates should refer to the endpoint's current service and budget limits, with separate checks for the busiest route.

Do not require identical prose. Require the same operational outcome where sameness matters: correct escalation, valid schema, permitted tool, and arguments that identify the same order. Human reviewers can score a stratified sample of the remaining reply differences for policy compliance and usefulness, while automated checks cover every shadowed request.

Keep rollback mechanical. Put the model selection behind a server-side configuration flag, preserve the previous prompt and tool schema, and retain the old route until the provider's retirement makes that impossible. During cutover, raise replacement traffic in stages while continuing the same measurements. If a gate fails, move traffic back without shipping application code.

The last check returns to the environment variable. Change it in one controlled route, confirm that logs show the intended resolved model, verify the shadow recorder cannot reach the real refund action, and watch the first production slice against the same gates used in rehearsal. The migration is complete only after the alias registry, fallback route, batch jobs, and runbook all name supported endpoints.

Questions people ask

How early should model API migration testing begin?

Begin when the provider documents a retirement or when your chosen alias is scheduled to move, rather than waiting for the final reminder. The useful duration depends on traffic: the test must cover a representative business cycle and leave time to revise prompts, tool schemas, or application validation before the old endpoint becomes unavailable.

Can stored test prompts replace shadow traffic?

No. A replay set gives both models identical inputs and makes regression testing repeatable, but it reflects the cases selected when the set was built. Shadow traffic captures current language, input lengths, and tool paths. Use redacted replays for controlled comparisons and sampled live traffic for coverage, subject to the same retention and privacy rules as production.

Should the replacement model execute tools during testing?

It should select tools and produce arguments, but write actions should terminate in a recorder or sandbox. Read-only results can be cached and shared between models. This preserves the evidence needed to compare tool behavior without sending duplicate messages, changing records, or issuing a second refund.

What if the provider recommends a direct replacement?

Treat the recommendation as the supported migration path, not proof of identical behavior. Run the successor through the same prompts, schemas, tools, and latency gates used by the current model. If it fails, the remaining time before retirement is what makes prompt changes, application fixes, or a different supported model practical.