What Is a Context Window?
A context window is everything a language model can see at once when it answers: system instructions, the entire conversation history, and any material you've attached. It works like a working memory that empties between sessions. When the window fills up, the earliest parts drop off or get truncated, and the model loses what it can no longer see.
A context window is everything a language model can see at the same time when it formulates an answer. It holds three things: the instructions that govern the model, the entire conversation so far, and the material you’ve fed in, like a pasted-in document. Outside the window, there’s nothing for the model to draw on.
The best analogy is a working memory. The model has no lasting memory of previous conversations – every session starts with an empty table, and on that table sits only what fits right now. That’s a crucial difference from how we ourselves remember things, and it explains a lot of how AI solutions behave in practice.
What Fits in the Window
The size is measured in tokens, chunks of text roughly the size of part of a word. Everything competes for the same space: the system instruction, every earlier exchange in the conversation, what you’re asking now, and the answer the model is in the process of writing. The more history accumulates, the less room is left.
That means a long conversation gradually fills the window. At the start there’s plenty of room, but after enough exchanges it starts getting cramped. And when it gets cramped, something has to give.
What Happens When the Window Fills Up
When the amount of text exceeds the window’s size, the oldest parts drop off, or get truncated. The model isn’t actively forgetting – it simply no longer sees what’s been pushed out. The distinction matters: the model doesn’t notice it itself and rarely signals that something has fallen away.
A concrete example. You ask a model to keep track of three requirements for an app while you reason through the details together. The conversation gets long. At the fiftieth message, you mention requirement one again – but the first requirement is no longer in the window, so the model answers as if it was never said. Nothing has gone wrong; the information is simply out of sight. That’s why long AI conversations sometimes feel forgetful: they’ve simply lost their own beginning.
Why Large Windows Don’t Make RAG Redundant
As windows grow, it’s tempting to think: just stuff in all the documents, and you can skip the hassle. In practice, that falls down on two points.
- The cost. The model charges per token, and the entire content gets sent along with every question. A large document sitting in the window gets paid for over and over, for every new exchange in the conversation.
- The precision. The more text the model has to search through, the easier it loses the right detail. A relevant passage drowns in a large mass, and the answers become vaguer rather than sharper.
| Approach | Cost per question |
|---|---|
| Everything in the context window | High – the entire material gets paid for every time |
| RAG retrieves relevant passages | Lower – only what's needed gets sent in |
The alternative is RAG, where the system retrieves just the passages relevant to the question and only sends those in. The window stays small and focused, cost stays down, and accuracy stays up. A large context window is convenient, then, but it doesn’t replace a well-thought-out method for selecting the right material. Filling a large window with everything is a bit like answering a question by reading the entire library – it works, but it’s expensive and easy to miss the right page.
Distinguishing It from Memory and Training
Two concepts often get mixed up with the context window, and it’s worth telling them apart.
One is the model’s training. What the model “knows” in general comes from its training and is baked into the model. The context window is something else: the temporary material it gets for this particular question. Training is the long-term memory, the window is the working memory of the moment.
The other is a built memory. If you want a service to remember a user between conversations, the window isn’t enough, since it empties out. In that case, you build a separate feature that saves relevant information and feeds it back in at the next conversation. The memory sits outside the model, then, not in the window.
What This Means for You as a Buyer
The context window is one of those concepts that explains why an AI solution needs more than just a good model. If the service needs to remember something over time, handle large documents, or keep operating costs down, it’s the architecture around the model that decides – not the size of the window itself.
A common pitfall is assuming a bigger window solves everything. In practice, it just pushes the problem ahead of you: the cost rises, and one day even the large window is full. The sustainable solution is almost always to select the right material instead of sending along everything.
A rule of thumb: assume the model only knows what’s currently sitting in the window, and build the solution accordingly. Want to know what a setup like that could look like for your needs? Get in touch with a description of what the service needs to handle.
Frequently asked questions
Is a context window measured in words or characters?
In tokens, which are chunks of text rather than whole words. A token roughly corresponds to a short word or part of a word, and Swedish text usually takes somewhat more tokens than English. When a vendor states a window of a certain number of tokens, that's the combined amount of instruction, history, and answer that has to fit inside it.
What happens when the context window fills up?
The oldest parts of the conversation drop off or get truncated to make room for the new. So the model isn't actively forgetting, it simply no longer sees what's been pushed out of the window. That's why a model in the middle of a long conversation can lose something you mentioned early on, without noticing it itself.
Does a large context window make RAG unnecessary?
No. Stuffing everything into a large window gets expensive, since cost follows the number of tokens, and precision drops when the model has to hunt for the answer in a large mass of text. RAG instead retrieves only the relevant passages for each question, which is usually both cheaper and more accurate.
Does the context window empty between different conversations?
Yes. Every new session effectively starts from zero, unless you deliberately feed in earlier information again. The model has no memory that persists between conversations on its own. If you want it to remember something over time, that has to be built as a separate feature, such as a memory store or a database.
Why do longer conversations get more expensive?
Because the entire history gets sent along again with every question. The longer the conversation gets, the more tokens the model processes each time, and the cost is calculated per token. A conversation that's grown long can therefore cost considerably more per answer than the first few exchanges, even if the question itself is short.