Your AI API Upgrade Needs a Rollback Plan

Consider one concrete workflow: a support assistant reads an order, calls `lookup_order`, decides whether policy permits a refund, calls `issue_refund`, then tells the customer what happened. The retiring model completes that chain today. Its designated successor may understand the same prompt and expose the same context window, yet call `issue_refund` with different arguments, refuse an allowed request, add slower deliberation, or write a confirmation that sounds less certain.

That difference matters more than a provider’s general capability description. The support assistant has one acceptable outcome: the right order receives the permitted refund once, and the customer sees an accurate confirmation. A migration plan should measure that outcome rather than assume the newer model is equivalent.

Providers document model retirement through different mechanisms. OpenAI maintains a deprecations page and offers model snapshots intended to keep behavior consistent. Anthropic documents deprecation states and retirement timing for Claude models. Google publishes model availability and lifecycle information for Gemini.

Those documents establish when an endpoint or model ID may disappear; they do not certify that a successor preserves the behavior of your prompt, tools, safety settings, or response parser.

Pin the model before testing its replacement

Start by finding the exact model identifier in production. A floating alias, meaning a name that can point to a provider-selected version, weakens the experiment because its behavior may change while the team compares results. Where the vendor offers a dated or otherwise fixed snapshot, pin it for both the incumbent and candidate.

Record more than the model name. Capture the system prompt revision, tool schemas, sampling settings, safety configuration, maximum output length, API or SDK version, retry policy, and any response-format constraint. For the refund assistant, the migration unit is the complete configuration that turns a customer message into a tool call, not the model ID alone.

Put that configuration in version control and attach its revision to every trace. A trace is the record of one request and its model responses, tool calls, timings, errors, and final customer output. If the candidate begins omitting the `reason_code` field from `issue_refund`, the trace must show which schema and prompt produced the omission.

This setup makes a clean rollback newly practical. Operations can route traffic back to the old configuration without reconstructing a prompt from a dashboard or discovering that an SDK upgrade changed serialization at the same time.

Shadow the refund workflow without issuing refunds

Traffic shadowing copies production requests to a candidate system while only the incumbent controls the customer response. It exposes the replacement to real phrasing, long order histories, malformed addresses, and policy edge cases that a hand-built evaluation set will miss.

Do not let the shadow path perform writes. The candidate may call `lookup_order` against a read-only replica or a sanitized recording, but `issue_refund` should terminate at a mock that validates the arguments and records the intended action. Idempotency keys, which let a service reject a repeated operation, still belong on the live refund endpoint; shadow isolation should not depend on them.

Privacy rules remain unchanged just because the second response is hidden. Send the candidate only traffic that the provider and model are approved to process, preserve existing retention controls, and redact stored evaluation data where required. Shadowing also increases inference use because one customer turn can produce two model runs, so forecast the temporary API cost from actual input and output token volumes before copying all traffic.

Begin with recorded cases, then a small approved slice of live requests. Expand by workflow rather than by a random percentage alone. The ordinary “refund an unopened item” case may pass while exchanges, split shipments, prior credits, or requests written in another supported language fail. For the assistant, each slice should be labeled by policy branch so aggregate success does not hide a broken branch.

Compare the candidate at the action boundary. Exact wording rarely needs to match. The selected order ID, refund amount supplied by the order system, policy code, tool sequence, and final statement of status do.

Test the differences providers cannot normalize away

Tool calling is the first gate. A tool call is structured model output that asks application code to run a named function with specific arguments. Validate the candidate’s output against the same schema used in production, then check semantic correctness: a syntactically valid call can still select the wrong order or request a refund before the policy lookup finishes.

Run adversarial cases through the sequence. A customer may paste instructions asking the assistant to ignore policy, claim ownership of another order, or request two refunds in one message. The expected result should specify whether the model asks for clarification, declines, or hands off to a person. “Safe” is too vague to grade.

Safety behavior needs its own acceptance set because a successor may apply refusals differently even when the application prompt is unchanged. Include permitted cases near a restriction, such as a frustrated customer mentioning legal action while requesting an ordinary eligible refund. Over-refusal creates a support failure; under-refusal can authorize an action outside policy. Grade the disposition and customer explanation separately.

Latency should be measured end to end and by stage. Capture time to the first model output, time until a valid tool request, tool execution time, and time until the final answer. A candidate that produces prose quickly but delays the actionable tool call can lengthen the workflow even if its total token generation looks fast in an isolated benchmark. Compare tail latency, not only the median, because the slowest accepted requests determine timeout and retry behavior.

Output style belongs in the release gate when customers see it. For the refund assistant, require the final answer to state whether a refund was issued, identify the relevant order without exposing unnecessary data, and avoid promising a settlement date the payment service did not return. Do not grade tone with an impressionistic “better” score. Use observable failures such as unsupported promises, missing status, excessive repetition, or wording that claims success after the tool failed.

Turn acceptance into a release decision

Build a fixed evaluation set from scrubbed production traces plus known incidents. Keep every case’s expected action and permitted range of customer wording under review by the people who own support policy. The model team should not quietly redefine a failure as acceptable to make the migration pass.

Set gates before seeing the candidate’s full results. The replacement must never issue a write that the incumbent correctly avoids, must preserve required tool arguments, and must stay within the application’s existing timeout budget. Choose numerical thresholds from the service’s current baseline and risk tolerance rather than copying a generic target from a model benchmark.

The candidate does not need identical prose, and demanding exact text equality will reject harmless improvements. It does need equivalent business behavior. In the recurring refund case, a pass means the same eligible order reaches the same authorized action, no second refund occurs during retries, and the customer receives a statement supported by the tool result.

Review disagreements manually. Some reveal candidate regressions; others uncover incumbent errors or acceptance tests that encoded outdated policy. Label that third category instead of awarding either model a win. Otherwise, the migration can preserve a bug because the old output was treated as ground truth.

Roll out with two independent switches

Use one switch to select the model configuration and another to permit write-capable tools. First send a limited production cohort to the candidate while retaining the incumbent configuration. The application can let the candidate answer low-risk informational turns but route refund execution through the established path until tool-call results hold under live conditions.

Watch customer-visible and system-level signals together: tool validation failures, handoff rates, repeated requests, timeouts, provider errors, policy dispositions, and unsupported confirmation language. Segment them by model revision and workflow branch. A single overall error rate will not show that the candidate fails only after `lookup_order` returns multiple shipments.

Define rollback triggers in advance and make one operator able to activate them without deploying code. Rollback should restore the pinned incumbent model, prompt, schemas, SDK behavior, and safety settings as one configuration. Keep queued requests and retry workers in mind; they may continue using the candidate after the front door has switched back unless the model revision travels with each job.

A provider retirement date limits how long this fallback remains usable. Before that date, prepare a second fallback that does not depend on the retiring API: disable automated refunds, keep read-only order lookup, and hand the customer to an agent with the collected context. It costs more support time, but it prevents an untested successor from controlling money movement.

Once the old model is unavailable, retain its evaluation outputs, configuration, and traces according to policy. They remain the comparison record for later changes, even though they can no longer serve live traffic.

Questions people ask

Can

I test a replacement model without sending duplicate customer actions?

Yes. Copy the request to a shadow environment where read operations use approved data and write tools terminate at schema-validating mocks. Record the candidate’s intended refund call, but allow only the production path to reach `issue_refund`. Keep idempotency protection on the live endpoint as a separate safeguard.

Should

I use the provider’s latest model alias during migration?

Use a fixed model version or snapshot where the provider offers one. A floating alias can change during evaluation, which makes regressions harder to reproduce. Pin the incumbent and candidate alongside the prompt, tool schemas, SDK version, safety settings, and retry policy.

What if the successor is better overall but fails one tool workflow?

Do not move that workflow merely because broad evaluations improve. Route eligible tasks to the successor while the refund path stays on the incumbent, or disable automated writes and hand off to staff if retirement removes that option. Release decisions should follow the customer-visible action, not an aggregate model score.

What should happen after the old API is retired?

Keep the last validated configuration and traces for comparison, but use a fallback that does not call the retired model. For the support assistant, that can mean read-only order lookup followed by a human refund decision. The final customer message should describe the handoff rather than claim the refund succeeded.