RAG: From Chunks to Citations

Retrieval-augmented generation, RAG for short, is usually drawn as one box: a question goes in, an answer comes out. In reality it is a pipeline of six or seven steps, and most of its failures happen before the generator reads a single word.

The shape is fixed. Documents are parsed into plain text, cut into chunks, and indexed. A query is matched against that index, the candidates are merged and re-ranked into a short list, and the list is packed into the model's context window with a citation number on every passage. The model's only job is to ground an answer in the passages it was handed.

Chunking decides what can be found later. Chunks that are too large blur the embedding — the vector that stands for the chunk — and waste context space; chunks that are too small lose the sentence that explains them. Overlapping neighboring chunks is the standard patch: if a sentence straddles a boundary, some chunk still contains it whole.

Contextual retrieval pushes further. Before embedding, each chunk is prefixed with a short model-written summary of where it sits in the document. The chunk then carries its own context instead of relying on the search step to guess it.

Lexical retrieval scores word overlap. BM25 weights each query word by how rare it is across the documents (the inverse document frequency) and caps how much repeating a word can help, so a rare name or error code matches exactly while common words contribute little. It is fast, easy to inspect, and blind to paraphrase.

Dense retrieval turns the query and the chunks into vectors and compares them with cosine similarity, a measure of the angle between vectors where 1 means the same direction and 0 means unrelated. It matches meaning: how do I reset my password finds change your passphrase. It misses exact strings it never saw in training, like an order number.

Hybrid retrieval runs both methods and keeps both ranked lists. The two scores are not comparable — a BM25 score of 12 and a cosine of 0.71 do not share a scale — so the merge step works on positions instead. Positions are always comparable, and that is the whole trick.

The demo below makes the difference visible. Switch between BM25, dense, and hybrid on the same query, then move the fusion constant k and watch how much first place is trusted.

Reciprocal rank fusion is one line: RRF(d) = Σ 1/(k + rank_i(d)). For each list where document d appears, take its position (the first position counts as 0), add the constant k, and add up one divided by that number across all lists. A document near the top of either list scores well; a document near the top of both gets two contributions and usually wins.

The constant k, commonly 60, dampens the top of each list. A small k trusts first place heavily; a large k flattens the lists toward a simple vote. Because the fusion only reads order, it survives score drift between retriever versions, which is one reason it is the default merge in production.

Merging produces candidates, not a final order. A reranker reads the query and one chunk together, as a single piece of text, and produces a relevance score. Unlike the dense method above, which scores the query and each chunk separately, it can see how the words interact — which is exactly what separates a related passage from one that answers the question.

Reranking is the highest-value upgrade in the pipeline. Skipping it costs 10 to 30 points of recall@5 — the share of queries whose top five results contain the relevant passage. Adding it after contextual retrieval cuts retrieval failures roughly in half, with published stacks reporting 49 to 67 percent reductions.

Packing the context is a budget problem. Remove near-duplicate chunks, sort by rerank score, cut at a token limit, and attach an id to every passage. Those ids are what make citations possible: the model is told to tag each claim with its source, and the interface can turn that tag into a link to the exact passage.

Without passage ids, citations are decoration. With them, a reader can verify a claim in one click, which is the entire point of retrieval.

Not every query deserves an answer. If the best reranked score falls below a threshold, the system should say it does not know rather than let the model answer from memory. Answering from memory is how a RAG system hallucinates with a straight face.

The threshold is a calibrated number: real evidence must clear it, and questions with nothing relevant in the corpus must not. Refusal is a feature, not an admission of failure — a grounded 'I don't know' is worth more than a confident wrong answer.

Retrieval quality is measured before the generator enters the picture. Precision@k asks how many of the k returned chunks are relevant. Recall@k asks how many of the relevant chunks were returned at all. Mean reciprocal rank rewards putting the first relevant chunk high in the list. Chunk overlap and reranking are the two settings that move recall the most.

The problems below compute top-k selection, precision and recall at k, reciprocal rank fusion, chunk overlap coverage, and mean reciprocal rank.