Prompt Caching Makes Big Manuals Cheaper for Small AI Apps

A small support assistant often has one disproportionately large input: policy-manual.md, a static document covering refunds, account recovery, shipping exceptions, escalation rules, and the language an agent may use. Without caching, the model processes that manual again when each customer writes in, even if only the final message changed.

Prompt caching changes that request pattern. Anthropic introduced the feature in public beta in August 2024, while OpenAI announced automatic prompt caching in October 2024. Both implementations reward the same basic shape: a long, identical prefix followed by a shorter variable section.

That makes a full policy manual newly practical for applications whose traffic is clustered into support hours, test runs, or short bursts. It does not make every long prompt cheap, and a timestamp inserted in the wrong place can erase the benefit.

The model reuses a prefix, not an answer

Prompt caching stores intermediate computation from prompt processing. It is not a response cache, which would return an old completed answer without running the model again.

On the first request, the provider reads the support assistant’s instructions and policy manual normally, then records the reusable computation for the eligible prefix. On a matching request, the provider starts from that cached state, processes the new customer material, and generates a fresh answer. Output generation still happens every time, so output-token charges and generation time remain.

The distinction matters for accuracy. Two customers can ask about the same refund policy and receive answers based on their own orders because the conversation and order details sit after the cached manual. A cache hit should produce the same model behavior as ordinary prompt processing; its immediate effect is on input processing, billing, and time to first token, which is the delay before the model begins its response.

A miss is also uneventful. The provider processes the full prompt, may create a new cache entry, and returns the answer through the usual path. The application does not need a separate recovery model, although its latency and cost monitoring should distinguish hits from misses.

The request shape determines the hit rate

For the support assistant, the useful order is stable material first and volatile material last. The system instructions lead, policy-manual.md follows, and the cache boundary comes after the manual. Customer history, account data, retrieved records, and the latest message appear afterward.

That ordering creates an exact reusable prefix across customers. Anthropic’s prompt-caching documentation describes cache matching from the beginning of the prompt through a designated cache breakpoint.

OpenAI’s documentation likewise says cached prompts require exact prefix matches, although caching is automatic for eligible prompts of at least 1,024 tokens.

Now put `Current time: 9:42 AM` above the manual. The prefix changes on the next request, so content after that point cannot match the previous one. A customer name placed in the opening system message causes the same failure. Tool definitions can also sit before message content in a provider’s serialized prompt, which means changing a function description may invalidate the manual cache even though policy-manual.

md itself did not move.

Semantically harmless edits still count. Rephrasing one opening sentence, changing JSON serialization, or conditionally inserting an instruction before the manual can produce a different token sequence, which is the model’s encoded representation of the text. The cache compares that sequence rather than deciding whether two passages mean roughly the same thing.

Changes after the cached prefix are expected. The latest ticket can differ completely, and the assistant can still reuse the manual. Editing the manual itself should cause a miss, which is desirable when a refund deadline or escalation rule has changed.

The cost case is narrow but measurable

Anthropic’s current documentation prices a five-minute cache write at 1.25 times the model’s normal input-token rate and a cache read at 0.1 times that rate. A one-hour write costs twice the normal input rate.

The default five-minute entry is refreshed when it receives another hit, so steady traffic can keep the reusable prefix warm.

Those multipliers show why one repeated call can be enough to recover the write premium. For two requests sharing the same eligible prefix, the first five-minute write and one read cost 1.35 times the ordinary price of processing that prefix once, compared with 2 times without caching. That is a 32.

5% reduction for the repeated prefix. Across 10 requests, one write plus nine reads costs 2.15 times the single-pass rate instead of 10 times, a 78.5% reduction.

The calculation excludes every changing token after the boundary and all output tokens. If policy-manual.md accounts for most of the input, the request-level saving can approach the prefix saving. If a large customer transcript follows the manual, the total reduction will be smaller.

Latency follows a similar shape without a universal number. Reusing computation for a long manual can reduce time to first token, but generation speed after the first token does not receive the same benefit. Network delay, model load, tool calls, and a long answer can still dominate what the user experiences.

Low traffic weakens the case. If the assistant receives one request, then stays idle beyond the cache lifetime, Anthropic’s higher write rate can cost more than ordinary input processing. A rarely used internal bot may be better left uncached unless the provider applies caching automatically without a separate write premium.

Providers expose different controls

Anthropic lets developers mark cache boundaries and choose supported lifetimes, which gives the application control but makes prompt layout part of the implementation. Its API usage fields report cache creation and cache read tokens, allowing a team to separate cold writes from warm hits.

OpenAI automatically applies prompt caching to eligible requests on supported models. Developers still need the repeated prefix, and the response usage object reports cached tokens under prompt-token details. The practical work moves from selecting a cache boundary to maintaining deterministic ordering and checking that the reported cached-token count is substantial.

Google’s Gemini API offers explicit context caching, with model-dependent minimum input sizes, a chosen lifetime, and storage charges alongside cached-input charges. That arrangement suits an application that wants to create and reference a named cache, but a small team must include storage duration in its comparison rather than looking only at the discounted input rate.

Provider behavior is therefore not interchangeable. Before changing architecture, check the selected model’s eligibility, minimum prompt length, retention policy, input multipliers, and usage fields in the current documentation. A model switch can change the cache economics even when the support assistant’s code remains intact.

A safe rollout starts with one canonical manual

Load policy-manual.md from one versioned source and serialize it identically for every request. Keep timestamps, request IDs, customer attributes, and live records after the cached section. If the application supplies tools, hold their definitions stable where possible and treat a tool-schema edit as another likely cold start.

Then send the same prefix twice within the provider’s cache lifetime. The first request should report cache creation or no cached tokens, depending on the API; the second should report a cache read or a nonzero cached-token count. Record that count beside total input tokens and time to first token. A nominal hit on a small fraction of the prompt will not support the expected cost reduction.

Run one deliberate invalidation test by editing a sentence in the manual. That request should process the revised prefix normally, while subsequent requests should reuse the new version. If an old policy appears in an answer, inspect the application’s deployed file and request construction first: exact matching prevents a revised prefix from hitting the old cache, but it cannot stop the application from continuing to send an outdated manual.

Prompt caching also changes the alternative to consider. Retrieval-augmented generation, or RAG, selects relevant passages from an external index before each model call and can keep inputs shorter, though retrieval may omit a rule the answer needed. Caching the complete manual avoids that selection step when the document fits the model’s context window and enough requests share it. It does not solve document size, access control, or policy-quality problems.

For the support assistant, the adoption decision can come from one log: the share of input tokens reported as cached during a normal traffic window. If that share stays low, move volatile content behind the boundary before paying for more infrastructure.

Questions people ask

Does prompt caching return the same answer to different customers?

No. It reuses computation for the matching prefix, then processes each customer’s conversation and generates a new response. Customer-specific material must sit after the cached manual; placing it before the boundary both reduces reuse and risks mixing personalization into what the application intended to keep stable.

Does changing one word invalidate the whole cache?

A change invalidates matching from the point where the token sequence diverges. If that word is near the start of policy-manual.md, most of the intended cached prefix may miss. A change after the cache boundary does not invalidate the manual, which is why request IDs and current messages belong later.

Is prompt caching worthwhile for a low-traffic app?

Often not with a paid cache write and a short lifetime. The first write may cost more than ordinary input processing, and savings arrive only through subsequent hits. Measure requests that share a prefix within the documented lifetime, then compare the write, read, variable-input, and output charges.

Can prompt caching replace a vector database?

Only in a narrower case. A vector database supports RAG by selecting passages from material that may be too large or change frequently, while prompt caching works best when the same complete document fits in context and repeats across requests. The support assistant can skip retrieval only if sending the full manual remains accurate and manageable.