What Is RAG? Retrieval-Augmented Generation Explained

The Problem RAG Solves

LLMs like GPT-4 or Claude have a training cutoff — they know nothing about your internal documents, your product specs, your company policies, or events after their training date.

You have two options when you need a model that knows your data:

• Fine-tuning: Retrain the model on your data. Expensive, slow, doesn't update easily.
• RAG: Keep the model as-is, but at query time, retrieve relevant documents and inject them into the prompt. Fast, cheap, always up-to-date.

RAG is almost always the right choice for knowledge-base use cases.

How RAG Works: Step by Step

• Ingest: Your documents (PDFs, Notion pages, FAQs, CSVs) are chunked into small pieces.
• Embed: Each chunk is converted into a vector (a list of numbers) using an embedding model. Similar meaning = similar vectors.
• Store: Vectors are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant).
• Query: User asks a question. The question is also embedded into a vector.
• Retrieve: A similarity search finds the top-k most relevant document chunks.
• Augment: Retrieved chunks are injected into the LLM prompt as context.
• Generate: The LLM answers based on the retrieved context — not its training data.

A Concrete Example

Company: An HR team wants an AI that answers policy questions. Without RAG: "What is our leave encashment policy?" → Generic HR answer based on training data. With RAG: The system retrieves the actual leave policy PDF section, injects it, and answers with the company's specific rules, citing the exact page.

RAG vs Fine-Tuning

Common RAG Mistakes

• Chunk size too large: Stuffing entire documents into chunks drowns the relevant signal.
• No chunk overlap: Split at wrong boundaries, losing context. Use overlapping windows.
• Bad embedding model: Using a general embedder for domain-specific text (medical, legal) gives poor retrieval.
• Top-k too small: Retrieving only 1–2 chunks misses relevant context. Start with 5–10.
• No reranking: Vector similarity isn't always semantic relevance. Add a reranker (e.g., Cohere Rerank) for better results.
• Ignoring metadata filtering: Filter by date, department, or document type before similarity search for precision.

Build a RAG System for Your Business

In our AI workshop, you'll build a working RAG system over your own documents — live, hands-on, with real code.

Further reading

The original RAG research paper by Lewis and colleagues.

Anthropic's engineering discussion of retrieval quality and document context.