A Cheap AI Router Saves Nothing If It Misses Hard Requests

The revealing line in my test log looked reassuring: `route=cheap confidence=high escalation=false`. The request behind it was not reassuring at all: “I was charged twice and need one payment reversed before rent is due.”

I used that support ticket as the anchor for a hands-on evaluation of a two-tier pipeline. One lower-cost model handled requests classified as easy, while a stronger, more expensive model took requests that crossed an escalation threshold. Easy policy questions belonged on the cheap route. Messages that required interpreting account history, resolving ambiguity or drafting a response around a consequential billing problem belonged on the strong route.

The cheap model answered the double-charge ticket fluently. It summarized a generic refund policy and suggested waiting for processing. That was a failure, even though the prose was coherent, because the request needed account-specific investigation rather than a policy recital. The router’s high confidence made the mistake more dangerous and, from a cost perspective, more important: no second model was called, so the pipeline recorded a cheap success until an outcome-based review rejected it.

That distinction changes the economics. A router does not save money by sending many requests to the cheaper model. It saves money by sending requests there that the cheaper model can finish correctly.

The one-call baseline is harder to beat than it looks

The baseline for this workflow was one call to the stronger model. Routing added a decision before generation: inspect the request, estimate whether the cheaper model would fail, then select a tier. A lightweight rules engine can make that decision without another model call, but an LLM router, which uses a language model to classify the request, adds its own token charge and response time.

The cost equation is straightforward. If routing happens before generation, expected model cost per request is the router cost plus the cheap-model cost for accepted easy requests and the strong-model cost for escalated requests. If the cheap model answers first and a verifier escalates afterward, every escalated request pays for both generation calls, plus any verifier used between them.

That second design is common because it feels safer. It is also where the discount starts disappearing. The double-charge ticket consumed a cheap-model answer before the pipeline recognized that a stronger response was needed. The corrected result therefore required at least two generation calls rather than the baseline’s one, and the first answer had no reuse value because its central recommendation was wrong.

A routed system must beat the baseline on total cost per completed request, including retries, validation calls and rejected outputs. Comparing the two models’ input and output token rates answers a narrower question. It does not show whether the application spends less.

Confidence is not an escalation policy

The first failure came from treating model confidence as evidence. Self-reported confidence is the model’s own estimate that its answer is correct; it is useful only when calibrated, meaning that similarly scored answers fail at roughly predictable rates in the target workflow.

Fluent models can assign high confidence to responses built on the wrong interpretation. In the billing ticket, the cheap model recognized familiar words such as “charged,” “payment” and “reversed,” then mapped them to a refund-policy pattern. It did not reliably distinguish a general policy request from a possible duplicate transaction requiring record review.

A better router uses observable difficulty signals before it asks for an answer. For this queue, those signals included whether the request referred to a specific transaction, whether resolving it required information absent from the message, and whether the customer described a deadline or immediate financial consequence. Those features do not prove that the stronger model will succeed, but they expose why the cheap model may be an unsafe endpoint.

The escalation threshold should follow the cost of a missed failure. In compact form, route to the stronger model when the estimated probability of cheap-model failure, multiplied by the cost of that failure, exceeds the stronger model’s incremental price and latency penalty. The failure cost can include a human correction, another model call, customer follow-up or an operational error. It should not be reduced to token spend.

This makes routing workflow-specific. A cheap model may be acceptable for rewriting a greeting even with modest confidence, while the same score should trigger escalation for a disputed payment. One global confidence threshold cannot express both decisions well.

Duplicate calls turn caution into a surcharge

Post-generation verification catches some mistakes, but it changes the routing pipeline into a cascade. The cheap model writes an answer, a gate inspects it, and the strong model starts again if the gate rejects it. A cascade can improve quality, yet every correct escalation now carries the cheap call as overhead.

The accounting gets worse when the verifier is another language model. The application may make one call to classify the request, one to draft cheaply, another to judge the draft and a final call to replace it. That is four calls around one customer message, even though the original alternative was a single strong-model call.

Rules can keep the gate cheaper. In the test workflow, the application could escalate before generation whenever the message mentioned a specific charge but lacked transaction details. It could also reject drafts that offered a resolution without citing information present in the request. These checks are narrower than asking a model whether an answer “looks good,” which often rewards polish rather than task completion.

Agreement checks have the same economic problem. Asking two cheap models to answer and escalating when they disagree may detect unstable cases, but agreement is not correctness, and two calls set a higher spending floor. For the double-charge ticket, two models could easily converge on the same generic refund policy because both recognize the same superficial pattern.

Parallel calls trade money for speed. A system can start the cheap and strong models together, then keep the cheap result when the router approves it and cancel the other call. That removes some sequential delay, but the strong model may already have consumed billable input or generated output before cancellation. Parallel execution is hedging, not cheap routing.

Routing latency arrives before the first useful word

A pre-generation router sits on the critical path, the sequence of steps that must finish before the user receives a result. Even a small classification call delays time to first token because the selected model cannot begin until the route is known. A post-generation verifier moves part of the delay later, but rejected answers still wait through the cheap generation before the stronger model starts.

Average latency hides the cases that matter. Teams should inspect tail latency, especially the 95th percentile, which is the duration that 95 percent of requests finish within. Escalated requests tend to populate that slow end because they traverse more stages. If the support interface has a fixed timeout, a correct strong-model answer that arrives after the deadline is still an operational failure.

The double-charge ticket demonstrated the worst ordering: classify, generate cheaply, reject, then regenerate strongly. Each stage behaved as designed, yet the combined pipeline was slower and more expensive than sending the request directly to the stronger model. Routing had introduced a tax precisely where the workflow needed the most care.

Test the accept region, not the whole benchmark

A conventional benchmark reports how well each model performs across all test requests. A production router needs a different measurement: how often the cheap model is correct among the requests the router allows it to finish. That subset is the accept region.

I evaluated the support pipeline by replaying authored requests and preserving the route, confidence, generated answer and final disposition in one trace. The key review was not whether the strong model beat the cheap model overall. It was whether requests accepted by the cheap tier were genuinely complete, grounded in the supplied information and appropriate for the consequence described.

Run this evaluation in shadow mode first. Shadow mode records the router’s proposed choice without letting it control the live response, so every request can still follow the established baseline while the team measures hypothetical savings and misses. The temporary evaluation cost is higher if both paths run, but it exposes false confidence before customers do.

Labels should come from task outcomes or a reviewed rubric, not solely from another model’s preference. For the billing case, the rubric required the answer to recognize that the transaction needed investigation and to avoid claiming a refund status unsupported by the message. A polished policy summary failed those requirements.

Track the escalation rate beside accepted-request accuracy, end-to-end latency and total model cost per completed answer. Those measurements reveal different faults. A low escalation rate with weak accepted accuracy means the router is reckless. A high escalation rate with strong accuracy may be safe but economically pointless.

Rising completed-answer cost usually exposes duplicate calls that model-level price comparisons omit.

The deployment decision should be based on a cost curve rather than one threshold picked by intuition. Replay the same requests across several escalation thresholds, then compare the savings with the resulting increase in reviewed failures and slow responses. If no point improves on the one-call strong-model baseline at an acceptable error level, the router is not worth running yet.

Make escalation cheap before making inference cheap

The most useful redesign in this workflow was to move obvious risk checks ahead of generation. Requests tied to a specific payment, missing account evidence or carrying a material deadline went directly to the stronger tier. The cheap model kept the bounded work: policy lookup, tone changes and drafts whose required facts were already present.

That arrangement sends fewer requests through the bargain path, so its headline savings look smaller. The accepted outputs are more defensible, however, and an escalation costs one strong-model call rather than a discarded cheap answer followed by a replacement.

Keep the log line from the double-charge ticket. A routing trace should record why the system chose a tier, what evidence the decision used, whether another call followed and how the final answer was judged. `confidence=high` is not enough. The useful field is the documented reason that made a cheap completion safe to accept.

Questions people ask

When does model routing save money?

Routing saves money when a meaningful share of requests can be completed correctly by the cheaper model after paying any classification overhead. Compare total model spend per accepted result with a one-call strong-model baseline, and include verifier calls, retries and outputs discarded after escalation.

Should the cheap model decide when to escalate?

It can contribute a signal, but its self-reported confidence should not control the route alone. Use request metadata, required evidence and workflow-specific risk rules, then calibrate the threshold against reviewed outcomes. The double-charge ticket showed why fluent confidence can miss the distinction between policy lookup and account investigation.

Is it better to route before or after generation?

Pre-generation routing usually has better cost mechanics because an escalated request pays for one generation model. Post-generation routing can inspect the draft, but rejected cases pay for both models and wait for both. Use post-generation checks only when the draft exposes failure signals that the original request cannot reveal.

What should a routing log record?

Record the proposed tier, the evidence used, the confidence score, every model call, token usage where available, elapsed time and final disposition. For an escalated request, the trace should make the duplicate cost visible rather than attributing the completed answer only to the stronger model.