The Context Window Defined
Every LLM has a maximum context window — the total number of tokens (input + output combined) it can process in a single request. Think of it as the model's working memory: everything it can "see" at once.
- Early models (GPT-2, 2019): 1,024 tokens (~750 words)
- GPT-3.5 Turbo (2023): 16,384 tokens (~12,000 words)
- Claude 3 / Gemini 1.5 (2024): 200,000–1,000,000 tokens (~150,000–750,000 words)
If a conversation exceeds the context window, the model cannot process the overflow. Older frameworks would silently truncate the beginning of the conversation, causing the model to "forget" earlier context — the "goldfish memory" problem.
The Bigger Window Trap
Modern models with million-token context windows seem to solve the problem entirely. In practice, using the full window creates new problems:
-
Cost: Every token in the context is an input token you pay for. Sending 500,000 tokens of context with every question costs roughly $5 per query at typical pricing — unsustainable for most applications.
-
Latency: The prefill phase (processing input) scales with context length. A 100k-token context can take several seconds to prefill before the model generates its first output token. This is called Time to First Token (TTFT) latency, and it directly impacts user experience.
-
"Lost in the Middle" Degradation: Research has shown that LLMs perform significantly worse at retrieving information from the middle of very long contexts compared to information at the beginning or end. A model given a 100-page document may miss a critical fact on page 50 while correctly citing facts from page 1 and page 100.
RAG: The Practical Solution
Retrieval Augmented Generation (RAG) is the standard solution to context window economics. Instead of sending the entire knowledge base to the model, you:
- Pre-process documents into chunks (e.g., 512-token paragraphs).
- Embed each chunk into a vector (a numerical representation of its meaning) and store it in a vector database.
- When a user asks a question, embed the question and find the most semantically similar chunks in the vector database.
- Send only those relevant chunks (typically 3–10 pages worth) to the LLM along with the question.
The result: instead of sending 500 pages of context, you send 5 relevant pages. Costs drop by 99%, latency drops dramatically, and accuracy often improves because the model isn't distracted by irrelevant information.