Use a Smaller AI Model Until the Accuracy Breaks

In 2024, major providers formalized a model choice that had previously required more infrastructure: OpenAI released GPT-4o mini in July, Google introduced Gemini 1.5 Flash in May, and Anthropic released Claude 3 Haiku in March. Each sat below a flagship tier in price and intended capability, giving developers a supported way to trade some reasoning depth for lower cost and latency.

That change makes task-specific routing practical. A team no longer has to send an entire support workflow to one expensive model or maintain its own compressed model. It can assign routine stages to a smaller hosted model and escalate the difficult cases, provided the team measures where the cheaper tier starts making consequential mistakes.

Consider one support ticket throughout: “Order A18427 arrived damaged. I attached two photos and want a refund, not a replacement.” The system must classify the issue, extract the order number and requested remedy, summarize the case, then select the correct internal tool. Those four jobs look similar in a chat window.

They impose different accuracy requirements.

One ticket contains four model decisions

Classification maps text into a fixed category such as `damaged_item`. Extraction copies specified facts into fields, while summarization compresses the message without changing its meaning. Tool routing selects a function, such as `look_up_order`, and prepares the arguments that another system will execute.

A flagship model, meaning a provider’s highest-capability general model, may perform all four stages well. That does not make it the efficient default. If a small model labels the ticket correctly and returns valid fields on almost every relevant example, additional reasoning capacity buys little at those stages, while every request still pays the flagship’s higher token price and usually longer response time.

The calculation changes at tool routing. Choosing `issue_refund` instead of `look_up_order` can cross a permission boundary and produce an external action. Here, one wrong answer may cost more than thousands of correct classifications save. The right comparison is therefore task against task, not model against model.

Classification usually has the clearest small-model case

Classification becomes tractable when the label set is fixed, each label has a written definition, and the model can abstain rather than force every ticket into a category. The damaged-order ticket should produce one allowed label, not a paragraph explaining the choice.

Test the small and flagship models on the same reviewed tickets. Report macro-F1, a score that gives each category equal weight, because overall accuracy can hide weak performance on rare labels. Also inspect the confusion pairs that matter operationally. Mixing up `damaged_item` and `wrong_item` may send work to the wrong queue; mixing two internal reporting labels may have no customer effect.

A smaller model wins when its error rate remains inside the team’s accepted boundary and its responses arrive sooner or consume fewer billed tokens. It loses when labels overlap, policy language changes frequently, or the ticket depends on several messages whose relationship must be inferred. Those cases can be forwarded to the flagship instead of forcing one tier to handle 100 percent of traffic.

Extraction rewards strict schemas, not eloquence

For the example ticket, the desired output is short: `order_id` is `A18427`, `requested_action` is `refund`, and the attachment count is `2`. A structured output schema, which limits a response to named fields and allowed data types, removes much of the freedom that makes generation difficult to evaluate.

Measure exact match for identifiers, schema validity for the whole response, and field-level recall for facts the workflow cannot omit. Then split the results by messy conditions: identifiers with punctuation, forwarded email chains, contradictory requests and missing fields. A model that performs well on clean messages but guesses absent order numbers is unsuitable, even if its aggregate score looks strong.

The fallback should be explicit. If the identifier fails a format check or the requested action is missing, retrying the same prompt may reproduce the same mistake. Send the ticket to the flagship with the validation error attached, or ask a human for the missing fact. The small model remains useful because only failed records pay the extra step.

Summarization depends on what the summary controls

A two-sentence summary displayed beside the original ticket is a lower-risk job than a summary that replaces the source text in a claims review. The first can tolerate stylistic variation because an employee can open the message. The second cannot quietly drop that the customer requested a refund rather than a replacement.

Word-overlap scores are weak evidence here. A useful evaluation asks reviewers whether the summary preserves the requested remedy, the damaged condition and the existence of attachments, while introducing no unsupported fact. The small model may match the flagship on short, formulaic tickets, then fall behind on long threads with changing requests or disputed timelines.

Length also changes the economics. Summarization sends the full conversation into the model, so input tokens can dominate cost even when the output is brief. Route short threads to the smaller model, and use the flagship after the conversation exceeds a tested complexity boundary, such as multiple participants or conflicting instructions. Token count alone is an imperfect proxy; a long shipping notice can be easier than a short, ambiguous complaint.

Return to A18427. “Customer reports damaged order A18427, supplied two photos and requests a refund” is enough for queue display. If later messages show that the customer accepted a replacement, the system must summarize the whole thread rather than preserve the first request.

Tool routing needs a higher bar

Tool routing converts language into a system decision. The model might choose `look_up_order` and supply `A18427`, after which application code calls the order database. The model should not receive authority merely because it emitted a valid function name.

Start by separating read-only tools from tools that change records, send messages or move money. A small model can route routine lookups if the application validates every argument and rejects unavailable tools. A refund call should require policy checks and, where the consequence warrants it, human approval or a flagship review. Model size does not replace permissions.

Evaluate wrong-tool selections, invalid arguments and failures to abstain. Weight them by consequence instead of averaging them into one accuracy number. Selecting no tool for a valid ticket creates delay; selecting a refund tool for an ineligible order creates an external action. Those errors should not count the same.

For A18427, the initial route should usually be a lookup, not an immediate refund, because eligibility depends on order status and policy data absent from the ticket. A larger model may infer that sequence more reliably, but deterministic application code can enforce it every time by making `issue_refund` unavailable until the lookup succeeds.

Find the break point with a routing evaluation

Build one evaluation set from reviewed production examples, then freeze it while comparing models. Keep a separate holdout set for the final decision so repeated prompt edits do not tune the system to familiar tickets. Date-stamp both sets, record the exact model identifier and rerun them after a provider changes a model or the business changes its labels.

For each of the four stages, record task quality, end-to-end latency and cost per accepted result. Cost per accepted result includes retries and flagship fallbacks, not just the smaller model’s listed token rate. Latency should include the slow tail, commonly tracked as the 95th percentile, because a fast median can coexist with support tickets that stall during bursts.

Slice the results by the conditions most likely to fail. For this inbox, useful slices include multilingual messages, image references, long threads and conflicting requested remedies. Do not promote a model based on a slice containing only a handful of examples; gather more reviewed cases or keep that slice on the flagship.

The deployment pattern is a cascade, a sequence in which a cheap first model handles clear cases and a stronger model receives uncertain ones. Confidence scores from language models are not guaranteed probabilities, so calibrate the escalation rule against observed errors. Schema failure, missing required evidence and disagreement between two prompts can also trigger fallback without trusting a self-reported confidence number.

Run the small model in shadow mode first, where it produces answers without controlling the live workflow. Compare those answers with current outcomes. A staged rollout can then expose a limited share of traffic, with automatic rollback if the wrong-tool rate, extraction failures or tail latency crosses the preselected limit.

The resulting architecture may use the small model for classification and extraction, choose between tiers for summarization, and keep consequential tool calls behind the flagship or deterministic policy code. That split is more work than choosing one model from a leaderboard. It also shows exactly where the flagship earns its bill.

Questions people ask

Is a smaller AI model always faster and cheaper?

Usually within one provider’s product line, but deployment details can reverse the result. Retries, long prompts, overloaded endpoints and frequent flagship fallbacks add time and cost. Compare end-to-end latency and cost per accepted result on the same workload rather than relying only on listed token prices.

How much accuracy should I trade for lower cost?

Set the limit per task and consequence. A small decline on an internal display label may be acceptable, while any measurable increase in unauthorized tool calls may not be. Use reviewed production examples and weight errors by their operational effect instead of selecting one accuracy target for the whole workflow.

Can one prompt test classification, extraction and routing together?

It can, but a combined score hides which capability failed and makes fallback expensive because the entire request must be repeated. Separate outputs and evaluations let the system keep a correct extraction, rerun only the failed route, and assign different model tiers to each stage.

When should the flagship model remain the default?

Keep it where inputs are ambiguous, threads contain conflicting instructions, or an error can trigger a consequential action before validation. In the damaged-order workflow, a flagship review may be justified for policy interpretation, while a smaller model still handles the fixed label and order-number fields.