Context Windows Explained: The Working Set Is Not Memory

A context window is the maximum number of tokens a model can process in one request, including instructions, conversation history, tool definitions, retrieved documents, and the answer it generates. The important detail is that it is a working set, not a memory system.

That distinction explains most context-window confusion. A larger window lets you present more material at once, but it does not guarantee that the model will use every passage equally well, remember it on the next turn, or fit the requested answer after the input is loaded. The practical skill is not “put everything in the prompt.” It is deciding what deserves space, where it should appear, and how the result will be checked.

This article explains the mechanism with practical examples for chat, document analysis, coding agents, and retrieval-augmented generation. It also gives a compact budgeting rule you can use before sending a request.

The quick answer

What is a context window?

It is the token budget available to one model invocation. The budget is shared by the system prompt, user input, previous messages, tool schemas, retrieved content, and output reservation. If the request exceeds the limit, the API may reject it or remove older material, depending on the product and truncation policy.

Why does it matter?

Context controls what the model can condition its next token on. It affects cost, latency, truncation, tool reliability, and whether a response is grounded in the material you supplied. A 1-million-token limit changes what you can send, not what the model can flawlessly understand.

The non-obvious truth: Long context is closer to a larger desk than a better filing system. It helps when the relevant material is present and well arranged. It hurts when noise, duplicates, stale instructions, or irrelevant tool output crowd out the signal.

The foundations: tokens, inputs, and outputs

Models do not count words. They process tokens, which are pieces of text learned during training. For common English, OpenAI gives a useful rule of thumb: “one token generally corresponds to ~4 characters of text.” That is roughly three quarters of a word, though code, punctuation, non-English text, and unusual identifiers can tokenize very differently. The OpenAI tokenizer is the practical way to check a real prompt.

Suppose your model allows 128,000 tokens. That number is not a 128,000-token input allowance plus an unlimited answer. The request must fit inside the model’s total context budget. If you reserve 8,000 output tokens, your input, instructions, history, tools, and retrieved passages have roughly 120,000 tokens available. The exact accounting depends on the API, model, and hidden or provider-managed fields, so treat the remainder as a ceiling, not a target.

An API request usually contains five kinds of context:

1. Instructions: system and developer rules, format requirements, and safety constraints.
2. Conversation: previous user and assistant messages that the application resends or the provider stores.
3. Tools: function names, descriptions, parameter schemas, and sometimes tool results.
4. Evidence: files, search results, database records, code, or retrieved chunks.
5. Output room: the maximum response or reasoning budget you allow.

The model predicts the next token from this combined sequence. It does not create a durable database record just because a fact appeared in the prompt. If the fact is absent from the next request, it is absent from that request’s evidence unless the application stores and retrieves it.

What does a context window actually contain?

The short answer is that it contains the material available to one generation step, not a permanent transcript of everything the model has ever seen. A chat interface may make history feel persistent, but the application or provider must still serialize, summarize, cache, or retrieve that history for each turn. The window is the current request’s bounded working set.

That difference matters in a simple support bot. Imagine a conversation with 40 turns. On turn 41, the system may send the full history, a summary plus recent turns, or a provider-managed state object. Those are different engineering choices. “The model remembers” is an unsafe description unless you know which one is happening.

In an API, count more than visible prose. A tool definition can be hundreds or thousands of tokens. A tool result can contain a full JSON response, stack trace, or document dump. A screenshot or PDF may be converted into model-specific tokens. A developer message repeated on every call consumes space on every call.

OpenAI’s Realtime API documents the consequence directly: a 32,000-token context with a 4,096-token maximum output leaves 28,224 tokens for the conversation before truncation. Its default automatic truncation can remove the oldest messages, while disabled truncation returns an error instead. The Realtime truncation documentation is a useful mental model even when you use another API.

Why is “maximum context” not the same as “usable context”?

The maximum is a capacity number. Usable context is the portion the model can correctly locate, prioritize, and apply to the task. Research on Lost in the Middle found that models can perform well when relevant information appears near the beginning or end of long inputs, then degrade when the same information is placed in the middle. A larger container does not remove this retrieval problem.

The failure is easy to reproduce. Put a refund policy at the top of a 200-page support manual, add unrelated policies in the middle, and ask for the refund window. Then move the same policy into the middle without changing the question. If accuracy drops, the model has demonstrated a positioning weakness, not a token-limit failure.

This is why “stuff the whole repository into the prompt” is a poor default for coding agents. The repository may fit. The relevant call site, current tests, generated files, stale docs, and task constraints may still compete for attention. A smaller context containing the right files and the exact acceptance criteria often produces a more reliable change.

Google’s long-context guidance calls a context window “short term memory” and makes the upside clear: newer Gemini models support context windows of 1 million tokens or more, useful for large codebases, books, transcripts, and multimodal inputs. The same document still recommends organizing long inputs, placing important information where it can be found, and using context caching or retrieval when appropriate. Capacity expands the options. It does not erase information architecture.

How should you budget a context window?

Start with the output, then work backward. Reserve enough room for the answer, reasoning, tool calls, and formatting you actually need. Use the remainder for instructions, history, tools, and evidence. A practical formula is:

usable input = model limit - output budget - safety margin

For a 128,000-token model, a request might allocate 6,000 tokens to the answer, 4,000 to tool definitions and control instructions, and a 10% safety margin of 12,800 tokens. That leaves about 105,200 tokens for conversation and evidence. The numbers are illustrative, not a provider guarantee. Measure your own requests because tokenization varies.

The safety margin matters because a request that fits exactly is brittle. A longer user message, an extra tool error, or a few more retrieved chunks can push it over the limit. In production, alert when the working set crosses a threshold such as 70% or 80%, then summarize or retrieve before the provider has to truncate silently.

Here is a useful routing table:

When should you use retrieval instead of a larger window?

Use retrieval when the corpus is large, changing, or only partly relevant. Use long context when the task depends on relationships across a bounded set of material that you can present together, such as a single codebase slice, a contract package, or a long meeting transcript.

Retrieval is not a cheaper version of reading. It adds a new failure mode: the system can retrieve the wrong chunk, split an important definition, or miss a passage because the query was vague. But retrieval gives you control over the working set. You can log what was supplied, apply metadata filters, update stale records, and keep irrelevant material out.

Consider a legal-operations assistant with 500,000 policy tokens. A naive approach sends all policies and asks, “Can we approve this request?” A stronger pipeline retrieves the approval rule, the exception clauses, the latest regional policy, and the definition of “high risk.” It then asks the model to cite each decision to a supplied passage. The second prompt is smaller, auditable, and easier to test.

The practical rule is simple: if the answer depends on one or two facts, retrieve those facts. If it depends on interactions across a bounded document set, provide the set, but label it and structure it. Do not choose a huge context merely because the model accepts it.

Why do tool calls make context management harder?

Tools turn context into a growing event log. Every schema, argument, result, error, and retry can be fed back into the next model call. The agent may still fit inside the formal limit while losing the task’s original constraints inside repetitive operational detail.

An agent that searches a repository five times might receive five near-duplicate directory listings. A browser tool may return a full page when the agent needs one table row. A database query may include 10,000 records when the next decision needs three. The fix is not always a larger model. It is a smaller tool surface and a deliberate state representation.

For each tool, decide what survives the next turn. If you are building with an agent runtime, the Claude Agent SDK customer-support triage example is a useful adjacent case because its session and tool state make this boundary concrete:

• Keep the user’s goal and acceptance criteria stable.
• Keep the tool name, arguments, result status, and a compact result summary.
• Keep raw output only when the agent must quote, inspect, or transform it.
• Replace repeated output with a digest, pointer, or structured state.
• Mark failed attempts and stale results so the model does not treat them as current truth.

This is also where prompt caching can change the economics. Anthropic documents that long-context requests above 200,000 input tokens can use premium long-context rates for Claude Sonnet 4 when the 1M-token context window is enabled. The Anthropic pricing documentation shows why a context window is both a capability limit and a cost boundary. Repeating a large static prefix may be technically valid and financially wrong.

What does a good context layout look like?

A good layout makes the task, evidence, and constraints easy to separate. Put stable instructions in a stable place. Put the user’s current objective near the active request. Group evidence by source or claim. Label untrusted text as data, not instructions. End with the exact output contract and the conditions for refusing to guess.

For a document question, this pattern is usually enough:

TASK
Answer the question using only the supplied policy excerpts.
 
DEFINITIONS
...
 
EVIDENCE 1: Refund policy, updated 2026-05-14
...
 
EVIDENCE 2: Regional exception, updated 2026-06-02
...
 
DECISION RULE
If the excerpts conflict, report the conflict and cite both.
 
OUTPUT
Return: decision, cited evidence, uncertainty, next action.

For a coding task, include the issue, relevant files, current behavior, expected behavior, constraints, and tests. Put the desired diff boundary in plain language. “Change only the parser and add regression coverage” is more useful than “fix the bug in this repository.” That same separation is useful in a tool-access safety checklist, where the action policy should not be buried inside a large transcript.

The edge cases and breakages

A prompt can fit and still fail. Token limits say whether the request is accepted. They do not measure factual recall, instruction priority, or resistance to contradictory passages.

Summaries can erase the thing you need. A conversation summary may preserve the conclusion while dropping the condition that made the conclusion valid. Keep source links, identifiers, dates, and unresolved questions in structured state instead of trusting a free-form summary alone.

Code is unusually expensive. Identifiers, punctuation, indentation, and repeated boilerplate can tokenize inefficiently. A “small” repository by line count can be large by tokens, and generated files often add noise without adding reasoning value.

Multimodal inputs are not interchangeable with text. A provider may count images, audio, and PDFs differently from plain text. A 50-page PDF is not a stable token estimate until the provider’s ingestion path has processed it.

The oldest message is not always the least important. Automatic truncation often removes old content first, but system constraints, definitions, and user preferences may be older than the active turn and still essential. If you use automatic truncation, test which information disappears and add a durable summary or pinned state for it.

The honest take

The first limit is that context-window behavior is model-specific. Results from one model, tokenizer, API, or modality do not automatically generalize to another. Vendor documentation tells you capacity and mechanics. It rarely tells you the accuracy curve for your exact documents, language, tool chain, and task.

The second limit is that “lost in the middle” is not a universal constant. Better training, attention methods, ordering, retrieval, and task structure can reduce the effect. The useful conclusion is not “never send long context.” It is “test position, noise, and density instead of assuming a capacity headline predicts reliability.”

The third limit is economic. Long context can replace retrieval infrastructure for some workflows, but it can also increase input cost, latency, cache complexity, and privacy exposure. The right architecture depends on how often the corpus changes, how expensive a wrong answer is, and whether you need an audit trail.

A practical context-window workflow

Before shipping a prompt that depends on long context, run this five-step check:

1. Count it. Measure tokens for instructions, tools, history, evidence, and output reservation.
2. Cut it. Remove duplicates, stale logs, generated files, and evidence unrelated to the question.
3. Label it. Separate instructions from untrusted content and identify dates, sources, and conflicts.
4. Place it. Put the task and decision rule where the model can easily find them. Test relevant evidence at the beginning, middle, and end.
5. Verify it. Require citations, structured fields, tests, or a second retrieval pass. Log the actual context and outcome for evaluation.

For a support workflow, test 50 real questions with three layouts: full history, summary plus recent turns, and retrieved policy plus recent turns. Track answer accuracy, citation coverage, token usage, latency, and escalation rate. For a coding agent, test the same issue with the full repository and with a curated file set. Track first-pass test success and the number of irrelevant edits. This is the same reason AI benchmark results need workflow tests, rather than a single capacity score.

The bottom line

A context window is the model’s bounded working set for one request. It is not durable memory, a search index, or a guarantee that every supplied token receives equal attention. Bigger windows are valuable when the task depends on relationships across a bounded corpus. They are wasteful when used to avoid selecting evidence.

Use the smallest context that preserves the decision. Reserve output room. Keep stable state outside the transcript. Retrieve changing facts. Put important evidence in a testable layout. Then measure the actual task, because the model limit is only the first number that matters. If local inference is part of the constraint, hardware and model memory is the related decision, not a substitute for context evaluation.

FAQ

Is a context window the same as memory?

No. A context window is the input and output budget for a model invocation. Memory is an application feature that stores, summarizes, or retrieves information across invocations. A product may provide memory-like behavior, but it still has to place the relevant information into the current request.

Does a larger context window make answers more accurate?

Not automatically. It can improve accuracy when the missing evidence is present and the model can use it. It can reduce accuracy when extra material creates noise, conflicting instructions, or position-related retrieval failures. Accuracy must be measured on your task, not inferred from the maximum token count.

How many words fit in 100,000 tokens?

There is no fixed conversion. For common English, one token is roughly three quarters of a word, so 100,000 tokens is roughly 75,000 words. Code, tables, punctuation, non-English text, and document formatting can change the ratio substantially. Tokenize the actual input before relying on an estimate.

Should I send my entire codebase to an AI coding tool?

Usually not. Send the task, relevant call path, definitions, tests, and files that constrain the change. Add repository-wide context when the task genuinely depends on cross-cutting behavior. Compare both approaches on first-pass success, irrelevant edits, latency, and review effort.

When should I use RAG instead of long context?

Use RAG when the corpus is large, changing, or only partly relevant. Use long context when the task needs relationships across a bounded set of material and you can afford to present it together. In either case, log the evidence supplied and require the answer to cite or otherwise ground its claims.

What happens when the context limit is exceeded?

The API may reject the request, truncate older content, or apply a provider-specific compaction strategy. Do not assume which behavior occurred. Check the API configuration and logs, set an explicit budget, and preserve critical state in a structured summary or external store.

How do I test whether a model is using long context well?

Create a test set where the same answer appears at different positions, with varying distractor density and document order. Measure exact answer accuracy, citation correctness, refusal behavior, latency, and token cost. Include your real file formats and tool results, not only clean synthetic paragraphs.