Your Replacement AI Model Needs a Dress Rehearsal

A deprecation notice usually supplies two facts: the model that will disappear and the model the vendor recommends instead. It does not establish that the replacement will preserve your application’s behavior.

Consider one concrete workflow: a support assistant receives a refund request, retrieves the order, checks a policy document, drafts a response and, after human approval, calls an `issue_refund` tool with an order ID and amount. The model sits in the middle, deciding what to retrieve, whether to refuse, how to structure the tool arguments and when to ask for approval. A successor can complete the same prompt while changing any of those decisions.

That is why an API retirement needs a rehearsal rather than a calendar reminder. OpenAI, Anthropic and Google document model life cycles, retirement schedules and recommended migration targets, but their documentation also treats model versions as changing software rather than interchangeable infrastructure. The newly practical response is a shadow migration: run recorded production inputs through both models, prevent the candidate from taking real-world actions, then compare what the application would have done.

Freeze the workflow before testing the model

Start with the application version running today. Pin the current model identifier where the vendor permits it, record the system prompt, tool schemas, retrieval settings, sampling parameters and output parser, and place that bundle under version control. If the prompt changes halfway through the migration, the comparison stops isolating the model.

For the refund assistant, capture the request that enters the model and the evidence available at that moment: conversation history, retrieved policy passages, order status and tool definitions. Remove or tokenize personal data according to your existing handling rules. A replay set does not need every production conversation, but it must include the cases that create operational risk, including an eligible refund, an ineligible request, a missing order, conflicting policy text and an amount that requires approval.

Use at least 100 representative cases for an initial gate if your traffic and retention rules allow it, then add every distinct high-cost failure you already know about. The number is a setup target, not a statistical guarantee. A narrow workflow with five stable intents may become legible quickly; a multilingual assistant serving many policies needs a larger and more deliberately stratified sample.

Each case needs an expected application outcome. For the ordinary eligible request, that might be: retrieve the correct order, cite the controlling policy, propose the permitted amount and emit valid arguments for `issue_refund`, while leaving execution disabled. Exact wording should rarely be the pass condition. The workflow cares about the decision and action.

Run the successor in shadow mode

Shadow mode sends the same input to a candidate model without allowing its output to affect the user or external systems. Production continues using the old model. The candidate writes its response, tool requests, token counts, timing and errors to a separate log.

Block side effects at the tool layer, not in the prompt. A sentence telling the model not to issue a refund is weaker than replacing the production tool with a recorder that validates the proposed arguments and returns a synthetic result. The shadow assistant can then complete every step, including its response after the supposed tool call, without moving money or changing an order.

Replay fixed cases first. That makes failures reproducible and lets an engineer inspect the same refund request under both models. Once the candidate clears that set, duplicate a sample of live requests asynchronously, provided your vendor agreements and data controls permit it; live shadowing reveals traffic shapes that a hand-built set missed, although it adds a second inference call and therefore raises testing spend.

Keep concurrency controlled. If both calls compete for the same worker pool or vendor quota, the test can increase production latency even though the candidate never answers a customer. Separate queues and rate limits make the comparison less convenient, but they stop the migration test from becoming the outage.

Measure the refund, not the prose

Begin with task completion: did the candidate reach the right disposition for the refund case? Then inspect the decision points that produced it. Record whether retrieval selected the controlling document, whether a refusal was appropriate, whether the approval boundary remained intact and whether the proposed tool arguments passed the same schema validation used in production.

Refusals deserve their own label. A newer model may decline a request that the old model handled, or comply where your application expects a policy-based refusal, because safety behavior and instruction following can change across model generations. In the refund workflow, separate a model safety refusal from a business denial. The former may say it cannot assist; the latter should explain that the order falls outside the documented refund policy.

Tool behavior needs more than a valid-JSON check. A tool call, meaning a structured request from the model to application code, can be syntactically valid while selecting the wrong function, inventing an order ID or placing the full refund in a field meant for cents. Run candidate arguments through production validators and business rules, then compare the proposed action with the labeled outcome. Never let schema validity stand in for correctness.

Tokenization can also move the bill and the boundary. A tokenizer splits text into the units a model counts, and different model families may divide the same prompt differently. Log input and output tokens from the API response rather than estimating them from character count. Test your longest policy bundle in particular: a changed count can push retrieved evidence outside the context limit, trigger truncation in your client or increase cost even when the visible answer looks unchanged.

Latency should be measured end to end and by stage. Save the time to first usable output, total completion time, tool-call round trips and retry count, then compare percentiles rather than one average. The refund assistant may appear faster on short denials while taking longer on eligible cases that require retrieval and a tool proposal. The user experiences the branch they entered, not the fleet-wide mean.

Set gates before anyone sees the results

Write acceptance thresholds before running the candidate. For this workflow, deployment might require zero unauthorized refund proposals in the high-risk set, no increase in malformed tool arguments, a task-completion rate within an agreed margin of the baseline and latency within the support team’s service target. Choose figures from your own risk tolerance and current measurements; a vendor’s general benchmark cannot supply them.

Include cost per completed case, not only cost per token. A model with cheaper listed tokens may call retrieval twice, produce longer completions or retry malformed tool requests, while a more expensive model may finish in one pass. Divide total inference and tool-processing cost by successfully completed refund cases. That number connects the model change to the application budget.

Human review remains useful where labels are ambiguous, but blind the reviewer to model identity and give them a short rubric. Otherwise, polished phrasing can conceal a worse decision. Route disagreements back into the replay set, especially when the candidate found a legitimate outcome that the original label did not cover.

Roll out where rollback still works

Move a small production share only after the shadow gates pass. Keep the old path warm, preserve its prompt and parser, and route requests through a feature flag that can switch models without a client release. The rollback trigger should use application telemetry, such as unauthorized action proposals, parser failures or support escalations, rather than waiting for users to describe a vague decline in quality.

A rollback is real only if someone has exercised it. During the canary, send a test refund request through the candidate, switch the flag, verify that the old model receives the next request and confirm that queued tool actions retain their approval state. Model-specific response formats can otherwise leave work stranded during the switch.

The vendor’s shutdown date still matters. Set an internal decision deadline far enough ahead to investigate failures, change prompts or tools, rerun the replay set and complete the canary. If the candidate misses the gate near that deadline, the fallback may be a different supported model or a narrower workflow that routes uncertain refunds to a person. An endpoint that no longer exists is not a rollback plan.

Questions people ask

Is the vendor’s recommended replacement usually drop-in compatible?

Treat it as API-compatible only where the documentation says so, and behaviorally unproven until tested. The request may succeed while refusals, tool selection, token counts or output structure change enough to break the refund workflow’s approval and validation rules.

How large should a model migration test set be?

Start with at least 100 representative cases when traffic and data policy permit, then weight the set toward costly failures and uncommon branches. Coverage matters more than a round number: every refund state, approval boundary, tool error and policy conflict should appear before production rollout.

Can shadow testing double model costs?

For requests sent to both models, it adds a second inference charge and may add retrieval or processing costs. Limit live duplication to a controlled sample, use recorded cases first and calculate cost per completed workflow so extra tool calls and retries remain visible.

What should trigger an immediate rollback?

Use thresholds tied to the application: an unauthorized refund proposal, a failed approval boundary, malformed tool arguments above your set limit or a sustained breach of the latency target. Keep the old route callable until the candidate has cleared the canary and the rollback drill has succeeded.