A Million-Token Window Does Not Make Whole-Repo Analysis Cheap

Consider one concrete workflow: an engineer asks a model to trace an authentication change through a monorepo, identify affected tests, and draft a migration plan. After exclusions, the repository bundle measures 300,000 input tokens with the model vendor’s tokenizer. The engineer expects 20 follow-up turns while checking routes, middleware, configuration, and test coverage.

That 300,000-token bundle fits inside the million-token-class context windows documented for some models released in 2025. Fit is the least interesting number, though. If the application resends the bundle on every turn without a cache hit, the static repository text alone appears 6 million times in input-token accounting: 300,000 tokens multiplied by 20 requests. Questions, answers, tool results, and system instructions come on top.

This is the real whole-repository tax. A large window makes the request admissible. It does not make repeated input free, instantly processed, or equally useful.

The repository is not its file size

Models read tokens, the text units produced by a model-specific tokenizer, rather than bytes or lines of code. OpenAI’s documentation uses roughly four characters per token as a general English-language approximation, but code does not obey a dependable characters-per-token rule. Punctuation, indentation, long identifiers, encoded data, minified JavaScript, and model vocabulary all change the result.

A repository size shown by a file browser therefore cannot settle the question. The defensible procedure is to apply the same exclusions the production request will use, serialize the remaining files with their paths and separators, and run that exact payload through the vendor’s tokenizer or token-counting endpoint. Anthropic documents a token-counting API, OpenAI publishes tokenizer tooling, and Google exposes token-counting methods for Gemini.

The exclusions matter more than they first appear. Dependency directories, build output, source maps, vendored libraries, generated clients, snapshots, coverage reports, media, and lockfiles can consume a large share of a prompt while adding little evidence for an authentication migration. Removing them makes a larger repository newly practical without changing models.

Even useful files carry overhead. A bundle needs file paths and boundaries so the model can distinguish two identically named modules, while a chat request also includes instructions, prior messages, tool definitions, and the requested output. The advertised context window covers the combined request and response allowance according to each API’s rules, not just the source files.

Return to the 300,000-token monorepo. Its token count is a measured input, not a guess based on megabytes. That one setup step tells the team whether the repository fits, whether enough room remains for analysis, and which directories are inflating every call.

Repeated context is where the bill grows

Input pricing is usually quoted per million tokens and varies by model, tier, batch mode, and cache status, so the durable calculation is more useful than a price copied from a model card. For an uncached workflow, multiply the static repository tokens by the number of requests, then add changing instructions, conversation history, tool results, and output tokens at their respective rates.

For the worked case, 20 requests repeat six million static input tokens. Ten requests repeat three million. Shortening the initial question does almost nothing because the 300,000-token prefix dominates both totals.

This pattern catches applications that appear inexpensive in a one-shot playground test. Repository analysis tends to branch: the model requests a definition, the engineer corrects an assumption, a tool returns a failing test, and the next call carries the evidence forward. If the client reconstructs the full prompt each time, every branch can trigger another full-priced pass through the codebase.

Output limits remain separate. A migration plan with cited file paths may be modest, while generated patches or verbose repository summaries can add material output charges and extend completion time. Asking the model to summarize every file before addressing the task pays twice: once to ingest the files and again to emit a representation that the application could have built deterministically.

A cache discount has conditions

Prompt caching lets a provider reuse processing for an unchanged prompt prefix. It can lower billed input cost and time to first token, but vendor implementations differ enough that “cached” should never be treated as one portable feature.

OpenAI documents automatic prompt caching for eligible prompts and says cache hits depend on exact prefix matches. Anthropic documents explicit cache controls and cache breakpoints, with different prices for cache writes and reads. Google documents explicit context caching for Gemini, including a storage charge tied to the cached token count and cache duration. Model eligibility, minimum token counts, retention, and rates can change, so production estimates need the current page for the exact model.

All three approaches reward the same prompt layout: put stable material first and changing material later. In the monorepo workflow, the system instructions, tool definitions, repository map, and unchanged source bundle belong in the stable prefix. The engineer’s latest question, test output, and patch candidate follow it. Reordering files, changing a timestamp near the front, or modifying tool definitions can reduce or eliminate the reusable prefix even when most repository text is unchanged.

A cache hit is not zero work. The provider still bills cached reads under its own schedule, the application may pay to create or store the cache, and the model must attend to a large body of material while producing the answer. Caching is strongest when many requests reuse an identical, expensive prefix within the provider’s retention rules. It is weak for one-off reviews or repositories that change between nearly every call.

Teams should log four values per request: total input tokens, cached input tokens, output tokens, and time to first token. Without those fields, a lower invoice can conceal an interaction that still makes engineers wait.

Latency has two separate causes

Time to first token covers the delay before the model begins answering; generation latency covers the answer afterward. A large input can increase the first component because the service must accept, tokenize, and process more text. A long response increases the second. Network upload time also becomes visible when clients transmit a repository-sized payload on every request.

Vendors do not promise one universal latency curve across models, regions, load, and cache states. That makes a local measurement more credible than a generic seconds estimate. Run the same repository question with the full uncached bundle, a confirmed cache hit, and a selective-file prompt; record median and tail latency separately, because an interactive coding tool is often judged by its slowest routine turns rather than its best one.

The selective request will usually start with fewer tokens. It can still lose if retrieval requires several serial model calls, so the comparison must cover the completed task, not one API response. A code map generated ahead of time avoids much of that penalty.

Irrelevant files consume attention too

Context capacity is a storage limit, not a guarantee that every included detail influences the answer correctly. Vendor prompting guides consistently emphasize relevant context, clear structure, and deliberate placement. Anthropic’s long-context guidance, for example, recommends placing long documents before the query and identifying quoted evidence before completing a task.

A whole-repository dump works against relevance when the authentication change concerns six source files but the prompt also contains unrelated mobile clients, old migrations, fixtures, generated schemas, and duplicated implementations. The model may cite a deprecated path, blend two similarly named functions, or spend its answer explaining modules that never execute in the affected service. More context can reduce uncertainty about what exists while increasing ambiguity about what matters.

The fallback is evidence-driven expansion. Start the 300,000-token monorepo job with a code map: a compact index of paths, symbols, imports, ownership boundaries, and test relationships. Load the authentication entry points and their direct callers, then add configuration and tests when imports, references, or failing commands justify them. If the model identifies an unresolved symbol, fetch that definition rather than another broad directory.

Retrieval, which selects small context fragments for a particular query, works well when names and documentation match the task. Pure similarity search can miss indirect dependencies, so code retrieval should use language-server references, import graphs, search results, or repository metadata alongside embeddings. The model gets fewer files, but each file arrives for a documented reason.

Whole-repository loading still earns its cost for a first-pass inventory, a small codebase, or a cross-cutting review where the relevant boundary is unknown and the repository changes infrequently enough to support cache reuse. For repeated implementation work, selective loading usually wins. It lowers the uncached prefix, preserves room for test output and patches, and makes cited evidence easier to audit.

The adoption decision can be mechanical. Measure the serialized repository, remove generated material, estimate request count, verify cache behavior in logs, and compare one full-context run with one code-map run on the same task. If the full bundle does not improve file identification or patch quality enough to offset repeated input and waiting time, the million-token window is capacity the workflow does not need.

Questions people ask

How do

I estimate the cost before uploading a repository?

Serialize the files exactly as the application will send them, then use the chosen vendor’s tokenizer or token-counting endpoint. Multiply static tokens by expected requests, apply current uncached or cached input rates, and add changing messages plus output; do not estimate from repository bytes alone.

Does prompt caching make whole-codebase analysis cheap?

It can make repeated calls cheaper when the repository remains an identical prefix and requests arrive within the provider’s cache rules. Cache writes, reads, storage, prefix changes, and retention still matter, so confirm cached-token counts in production logs rather than assuming a repeated prompt hit the cache.

When should

I use retrieval instead of the full context window?

Use retrieval when a task concerns a bounded feature, service, error, or patch and the repository contains substantial unrelated or generated material. Pair text similarity with symbols, imports, references, and tests so indirect dependencies can trigger additional file loading.

When is sending the entire repository reasonable?

It is reasonable when the measured codebase fits with ample output room, the task may cross unknown boundaries, and several follow-ups can reuse a stable cached prefix. Exclude dependencies and generated artifacts first, then compare task-level latency and answer quality against a code-map workflow.