How RAG works
A RAG system has two parts.
Before anyone asks: indexing
Documents are collected from where they live, such as a document store, a wiki or a ticket system. They are split into chunks, usually no more than a few hundred tokens each. Each chunk is turned into an embedding, a list of numbers that captures its meaning, and stored in a search index, often next to a keyword index.
When someone asks: retrieval and generation
The question is searched against the index. The best-matching passages are retrieved, often re-ranked by a second model, and placed in the prompt with the question and instructions such as “answer only from these passages and cite them”. The model writes the answer, and the application shows the sources next to it.
RAG architecture options
| Option | How it works | Where it fits |
|---|---|---|
| Vector search | Finds passages close in meaning to the question, using embeddings | Questions worded differently from the documents. It can miss exact terms such as codes and names |
| Hybrid search | Runs keyword search (BM25) and vector search together and merges the results | A strong default for company content, which mixes prose with codes, names and numbers |
| Re-ranking | A second model reorders the top results by relevance | Large collections where the first search returns many near misses |
| Contextual retrieval | Adds a short description of its document to each chunk before indexing | Chunks that lose their meaning out of context, such as a clause or a table row |
| Long context | Puts whole documents into the model’s context window instead of searching | Small, stable sets of documents. It costs more per question |
In Anthropic’s tests, contextual embeddings combined with BM25 keyword search and re-ranking cut the rate at which the right passage was missing from the top 20 results by 67%. Azure AI Search, for example, runs keyword and vector queries in parallel and merges them with Reciprocal Rank Fusion, a common way to build hybrid search.
Before building an index, check what your AI platform already offers. Claude, for example, can work with connected sources such as Google Drive or Microsoft 365 through connectors, which is often enough for a first step.
Limits of RAG and how to manage them
- Missed passages. Splitting documents can cut a passage off from the context that gives it meaning, and embeddings can miss exact matches that keyword search finds. Hybrid search and contextual retrieval reduce both problems.
- Stale content. An index is a copy. When documents change, they and their embeddings have to be updated, ideally automatically and in the background.
- Permissions. The index must respect who may see what, so that a person only retrieves passages they are allowed to read.
- Answers that go beyond the sources. RAG reduces invented answers but does not eliminate them. Instructions to answer only from the passages, citations people can check and a plain “I don’t know” when nothing relevant is found keep answers honest.
- Too much context. Models can lose information placed in the middle of a long prompt, so fewer, better passages often beat more of them.
The way to know whether a RAG system works is to test it: a set of real questions with known answers and sources, measured for both retrieval and answer quality, before and after every change.
RAG, fine-tuning and long context
| Approach | What it changes | Use it for |
|---|---|---|
| RAG | What the model can see when it answers | Knowledge that changes, is large or needs citations |
| Fine-tuning | How the model behaves, through extra training | A consistent style, format or task, not facts that change |
| Long context | How much the model reads in one request | A few documents at a time, such as one contract or one report |
They combine well: a model can be tuned for a format, use RAG for knowledge and read a whole document when one is attached.
Where companies use RAG
- Policies and HR: answers on leave, expenses or security rules, with a link to the policy.
- Customer support: replies drafted from the knowledge base and past tickets, for a person to send.
- Legal and compliance: research across contracts, clauses and regulations, with every passage cited.
- Technical documentation: answers for engineers and field teams from manuals and specifications.
- Sales: product facts and text from past proposals, for a specific opportunity.
Sources
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020
- Anthropic: Introducing Contextual Retrieval, 19 September 2024
- Microsoft: Hybrid search in Azure AI Search
- AWS: What is retrieval-augmented generation?
- Liu et al.: Lost in the Middle: How Language Models Use Long Contexts