RAG Architecture: The "Open-Book" Test

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is the architectural pattern that combines vector databases with LLMs to produce accurate, grounded answers from private data. It is the dominant technique for building AI applications on top of proprietary knowledge bases.

The analogy is an open-book exam. A student who must rely purely on memory (a standard LLM) may guess when they forget something. A student allowed to bring a textbook (RAG) can look up the relevant passage before answering — producing a more accurate, verifiable response.

RAG has two distinct phases: ingestion (done once, or periodically) and query (done on every user request).

Phase 1: Ingestion (Preparation)

This phase runs before any user queries arrive. It prepares your knowledge base for fast retrieval:

1. Load documents: Collect your source data — PDFs, web pages, database records, Confluence pages, Slack messages, etc.
2. Chunk: Split each document into smaller, overlapping segments (typically 256–512 tokens each). Chunking is critical: too large and the retrieved context is noisy; too small and individual chunks lack enough context to be useful.
3. Embed: Pass each chunk through an embedding model to produce a vector.
4. Store: Write each vector — along with the original text and metadata (source URL, document ID, timestamp) — into the vector database. The database builds an index for fast retrieval.

Phase 2: Query (Runtime)

This phase runs on every user request:

1. Embed the query: Convert the user's question into a vector using the same embedding model used during ingestion.
2. Retrieve: Send the query vector to the vector database. Ask for the top-K most similar vectors (typically K=3 to 10). The database returns the matching text chunks.
3. Augment: Construct a prompt that includes the retrieved chunks as context, followed by the user's original question.
4. Generate: Send the augmented prompt to the LLM. The LLM reads the provided context and generates an answer grounded in your actual data.

A concrete prompt template looks like this:

You are a helpful assistant. Answer the user's question using ONLY 
the context provided below. If the answer is not in the context, 
say "I don't know."

Context:

---

[Retrieved chunk 1 text]

---

[Retrieved chunk 2 text]

---

[Retrieved chunk 3 text]

---

User Question: What is the company's parental leave policy?

Why RAG Beats Fine-Tuning for Most Use Cases

A common question is: "Why not just fine-tune the LLM on your data instead?" Fine-tuning trains the model's weights to internalize new knowledge — but it is expensive (hours of GPU time, thousands of dollars), requires re-training every time your data changes, and still doesn't eliminate hallucination. RAG is cheaper, updatable in real time (just re-embed new documents), and produces answers that are directly traceable to source documents. For most enterprise use cases, RAG is the right first choice. Fine-tuning is better suited for changing the model's behavior or style, not for injecting factual knowledge.