Start practising

AI ENGINEERING · AGENTS · KNOWLEDGE · 4 OF 5

Retrieval-Augmented Generation (RAG): Grounding Answers in Your Data

How retrieval-augmented generation grounds a model in your own documents: ingesting and chunking, embeddings and vector search, hybrid keyword plus vector retrieval, reranking, building a prompt with citations, rewriting queries and retrieval as an agent tool, freshness and permissions, measuring retrieval and faithfulness, common failures, and when to pick RAG over long context or fine-tuning.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
1 / 17

Press play to hear the narration, or step through at your own pace.

Notes

The problem

Here is the problem. The model never saw your private documents. Its knowledge stops at its training date. And when it does not know, it often fills the gap with something plausible, which is the definition of a hallucination. It also cannot point to a source. You could retrain the model every time a policy changes, but that is slow and expensive. There is a much simpler fix.

Why a bare model fails
  • • never saw your private docs
  • • knowledge frozen at training
  • • fills gaps with plausible
  • guesses (hallucination)
  • • cannot cite a source
  • retraining for every policy
  • change is slow and costly

The idea

The idea is an open book exam. Take the question, search your own documents for the most relevant passages, put those passages into the prompt, and ask the model to answer only from them and cite which one it used. Update a document and the very next answer reflects it, with no training at all. Two halves, then: an offline pipeline that prepares documents for search, and an online path that answers questions.

Open-book answering
  • 1 take the question
  • 2 search your documents
  • 3 put the best passages
  • into the prompt
  • 4 ask the model to answer
  • only from them, and cite
  • update a doc → the next
  • answer changes. No training.

Ingest

First the offline half, the ingest pipeline. Load documents from wherever they live. Clean them, stripping navigation menus and footers, fixing tables and keeping headings. Split them into chunks. Turn each chunk into a vector with an embedding model. Then store the vector, the text and metadata such as source, date and who may read it. Run this again whenever a document changes, ideally only for the changed parts.

Ingest pipeline
  • load: PDFs, pages, tickets
  • clean: strip menus, footers,
  • fix tables, keep headings
  • chunk: split into passages
  • embed: one vector per chunk
  • index: vectors + text +
  • metadata (source, date,
  • team, access list)

Chunking

Chunking matters more than people expect. A chunk should hold one idea. Too big and its vector becomes a blurry average that matches nothing well. Too small and the passage lacks the context to answer. Typical sizes are a few hundred tokens, with a little overlap so a sentence cut at a border is not lost. Split on the document's own structure first, its headings and sections, and prefix each chunk with its title so it makes sense on its own.

Chunking choices
  • size: ~200-800 tokens
  • overlap: ~10-20% so ideas
  • cut at a border survive
  • split on structure first:
  • headings, sections, rows
  • prefix each chunk with its
  • title and heading path
  • too big: blurry matches
  • too small: missing context

Embeddings

An embedding model turns a piece of text into a list of numbers, a vector, usually a few hundred to a few thousand long. Texts with similar meaning land close together, measured by cosine similarity. So cancel my plan sits near end my subscription, even though they share no words. That is what makes semantic search work. One rule: embed documents and queries with the same model, and re-embed everything if you switch models.

Embeddings
  • text → vector of ~384 to
  • 3,072 numbers
  • similar meaning → nearby
  • vectors (cosine similarity)
  • "cancel my plan" is close to
  • "end subscription" even with
  • no shared words
  • same model for docs + queries

Vector search

At query time, embed the question and find the nearest chunk vectors. Comparing against every vector is too slow at millions of chunks, so vector indexes use approximate nearest neighbour structures, like H N S W graphs, that find almost all the true neighbours in a few milliseconds. You do not need an exotic database. Postgres with the pgvector extension handles millions of vectors, and most search engines now support vectors too.

Finding neighbours fast
  • exact search compares every
  • vector: too slow at millions
  • ANN index (HNSW, IVF):
  • ~95-99% recall, ms latency
  • options: pgvector in
  • Postgres, a search engine,
  • or a dedicated vector DB
  • filter by metadata too

Hybrid search

Vectors are great at meaning but weak at exact strings: error codes, product names, part numbers. Classic keyword search, such as B M twenty five, is the opposite. So run both and merge the results. A simple, robust merge is reciprocal rank fusion, where each document scores by how high it ranks in each list. Hybrid search almost always beats either one alone, and it is cheap to add.

Keyword + vector
  • vectors: good at meaning,
  • weak at exact codes, names,
  • part numbers, rare words
  • keywords (BM25): the reverse
  • run both, merge the lists
  • (reciprocal rank fusion:
  • score = sum of 1/(60+rank))
  • hybrid usually beats either

Reranking

Search returns candidates, not answers. A reranker improves the order. Stage one is fast and wide, hybrid search for the top fifty. Stage two is a cross encoder model that reads the question and each candidate together and scores how well that passage answers it. Keep the top three to eight. Reranking adds maybe a tenth of a second, and it is often the single biggest quality improvement you can make.

Two-stage retrieval
  • stage 1: fast, wide
  • hybrid search → top 50
  • stage 2: slow, precise
  • cross-encoder scores each
  • (question, chunk) pair
  • → keep top 3-8
  • adds ~50-200 ms
  • often the biggest quality win

The prompt

Now build the prompt. Tell the model to answer only from the sources, to cite them by number, and to say so when the sources do not contain the answer. Then list the passages, each with an id, a title and a date, so it can tell an old policy from a new one. Put the question last. Citations let users check the answer and let you check the model. Permission to say I do not know is what cuts hallucinations most.

Grounded prompt
Answer using ONLY the sources.
Cite like [2]. If the sources
do not answer it, say so.

[1] Refunds (EU), 2026-08:
    "...within 14 days..."
[2] Refunds (US), 2026-05:
    "...within 30 days..."

Question: EU refund window?

Better queries

The question as typed is often a poor search query. A follow up like and for the E U means nothing alone, so have a model rewrite it into a standalone query using the conversation. In an agent, go further and make search a tool. The agent can search, read the results, notice something missing, refine the query and search again before answering. That handles multi part questions a single lookup would miss.

Query side
  • follow-up: "and for the EU?"
  • → rewrite with history:
  • "refund window for EU
  • customers"
  • agentic RAG: search is a
  • tool; the agent may search,
  • read, refine and search
  • again before answering

Freshness and access

Two things keep RAG correct over time. Freshness: re-ingest when a document changes, triggered by change events rather than a weekly batch, and delete chunks when the source is deleted, or the model will happily quote a retired policy. And access control: store who may read each chunk and filter at query time by the asking user. Never retrieve restricted text and hope the prompt keeps it hidden.

Keep it right
  • freshness: re-ingest on
  • change events, not weekly
  • deletes: remove chunks when
  • the source is removed
  • access: store who may read
  • each chunk; filter at query
  • time by the asking user
  • never rely on the prompt
  • to hide restricted text

Measuring it

Measure the two halves separately, or you will not know what to fix. For retrieval, build a set of questions with the passages that answer them, and measure recall at k: is the right chunk in the top five? For generation, measure faithfulness, meaning every claim is supported by the retrieved sources, along with answer relevance and whether the citations are correct. A model can grade these at scale once you have checked it against human labels.

Two things to measure
  • retrieval (needs labels):
  • recall at k: is the right
  • chunk in the top 5?
  • MRR: how high does it rank?
  • generation:
  • faithfulness: claims backed
  • by the sources?
  • answer relevance, citation
  • accuracy

Failure modes

When answers are wrong, look at retrieval first. The right document was a scanned P D F that never got its text extracted. The right chunk was found but ranked ninth. An old and a new version of a policy were both retrieved. The answer was spread over ten chunks. Or the model ignored the sources and answered from memory. Log the retrieved chunks with every answer, so you can tell these apart quickly.

What goes wrong
  • • right doc never chunked
  • (scanned PDF, table lost)
  • • right chunk ranked 9th
  • • stale and new versions
  • both retrieved
  • • answer spread over many
  • chunks
  • • model ignores sources and
  • uses its own memory

RAG or not

When should you use RAG? When the knowledge is large, changes often, or needs citations and per user access control. If the whole source fits comfortably in the context window, like one contract, just paste it in. Fine tuning is good for teaching a style, a format or a narrow skill, but it is a poor way to add facts and cannot be updated quickly. Most products start with RAG and a well written prompt.

Pick the tool
  • RAG: large or changing
  • knowledge, need citations
  • and access control
  • long context: one or a few
  • docs that fit; just paste
  • fine-tuning: style, format,
  • narrow skills; not a good
  • way to add facts
  • often: RAG + a good prompt

Say it like this

How to say it. Chunk by heading at around five hundred tokens with the title on each chunk. Hybrid keyword and vector search for the top fifty, then a cross encoder reranks to five. The prompt says answer only from the sources, cite them, and say when they do not cover it. Results are filtered by the user's permissions. We track recall at five and faithfulness on a labelled set before every release.

In the interview
  • "Chunk by heading, ~500
  • tokens. Hybrid search, top
  • 50, cross-encoder rerank to
  • 5. Prompt says answer only
  • from sources and cite. ACL
  • filter per user. We track
  • recall@5 and faithfulness on
  • a labelled set each release."

Interview questions

What is retrieval-augmented generation?

A pattern where the system first retrieves passages relevant to the question from your own data, adds them to the prompt, and asks the model to answer only from them, usually with citations. It grounds answers in current, private knowledge without retraining.

How do you choose a chunk size?

Aim for one idea per chunk, often a few hundred tokens with a little overlap, split on headings and sections first. Large chunks produce blurry embeddings; tiny ones lack context. Tune it by measuring retrieval recall on real questions.

What is an embedding and why does it enable semantic search?

An embedding is a vector produced by a model so that texts with similar meaning are close together. Searching for nearest vectors finds passages that match the meaning of the question even when they share no words.

Why use hybrid search instead of vectors alone?

Vector search captures meaning but misses exact tokens such as error codes, names and identifiers; keyword search like BM25 catches those. Merging both lists, for example with reciprocal rank fusion, usually beats either alone.

What does a reranker do?

It takes the top candidates from a fast search and scores each one together with the question using a more accurate model such as a cross-encoder, then keeps the best few. It costs tens to hundreds of milliseconds and often gives the largest quality gain.

How do you reduce hallucinations in a RAG system?

Retrieve better (hybrid search, reranking, good chunks), instruct the model to answer only from the provided sources and cite them, explicitly allow it to say the answer is not in the sources, and evaluate faithfulness regularly.

How do you enforce document permissions in RAG?

Store access metadata with every chunk and filter retrieval by the requesting user before anything reaches the prompt. Never rely on instructions in the prompt to hide text the user should not see.

How do you evaluate a RAG pipeline?

Measure retrieval with a labelled question set (recall at k, mean reciprocal rank) and generation with faithfulness, answer relevance and citation accuracy. Measuring them separately tells you which half to fix.

When would you use RAG instead of fine-tuning?

Use RAG for knowledge that is large, changes often or needs citations and access control. Fine-tuning suits style, output format or narrow skills, but is a poor way to add facts because they cannot be updated or cited easily.