What is a context window in an LLM?

Every request to a language model is a single block of tokens. The context window is the size limit on that block. It is the model's entire working memory for one request: if a fact is inside the window, the model can use it; if it is not, the model cannot know it, however important it was in an earlier message.

What fills the context window

A typical chat or RAG request contains several parts, all counted against the same limit:

The last row surprises people. The output shares the window with the input. If a model has a window of 128,000 tokens and your prompt is 126,000 tokens, it can write at most about 2,000 tokens before it hits the limit. Many APIs also set a separate, smaller maximum for output tokens.

Models do not remember between requests

A language model is stateless. When a chat app seems to remember what you said ten messages ago, it is because the app sends those ten messages again, inside the context window, with every new question.

That has two consequences:

1. Long chats get more expensive every turn, because the history is re-sent and re-billed each time.
2. Old messages eventually fall out. When the history no longer fits, the app must drop, truncate or summarise older turns. Whatever gets dropped is gone for the model.

What happens when you exceed the window

Depending on the provider and the app, one of these happens:

• The API rejects the request with an error about the maximum context length.
• The app silently truncates, usually by dropping the oldest messages or the middle of a document.
• The answer is cut off when the output reaches the remaining space.

Silent truncation is the dangerous one. The model answers confidently from whatever is left, and you may not notice that a key paragraph never reached it.

Budgeting a context window

Treat the window as a budget and plan it before you write code. A simple plan for a RAG tutor might look like this:

Window                         32,000 tokens
- System instructions             600
- Conversation history (capped) 4,000
- Retrieved chunks (5 x 400)    2,000
- Current question                200
- Reserved for the answer       1,500
= Used                          8,300   (plenty of headroom)

Capping each part keeps the request predictable. The tutor on this site does the same: it limits how much history it sends and how many retrieved chunks it includes, and it shows you how many tokens each answer used.

Useful habits:

1. Count, do not guess. Use the model's tokenizer. Try the token counter to see how quickly text fills a window.
2. Reserve room for the answer before adding documents.
3. Summarise or trim history instead of sending every past message.
4. Retrieve instead of stuffing. Sending the five most relevant passages usually beats sending the whole manual. That idea is the heart of retrieval-augmented generation.

Bigger is not automatically better

Context windows have grown from a few thousand tokens in early models to hundreds of thousands, and some models now accept a million or more. That is useful for whole-codebase or whole-book tasks, but it comes with trade-offs:

• Cost and latency grow with input size. You pay for every token you send, on every request.
• Attention is spread thinner. Research has shown models can miss information placed in the middle of long inputs. See Why long context windows still miss things.
• More text means more noise. Irrelevant passages can distract the model or contain instructions you did not intend, which is a prompt injection risk.

Summary

• The context window is the maximum number of tokens one request can hold, input and output together.
• Models are stateless; apps create memory by re-sending text inside the window.
• Plan a token budget, reserve room for the answer, and retrieve relevant text rather than sending everything.