RAG Architecture
Grounding LLMs in your own data — chunking and indexing, embedding-based retrieval, reranking, prompt augmentation, and why retrieval quality decides everything.
Retrieval-Augmented Generation gives an LLM knowledge it wasn't trained on by retrieving relevant context at query time and injecting it into the prompt. An offline step chunks, embeds, and indexes your documents; at query time it embeds the question, retrieves (and reranks) the best chunks, and the model answers grounded in them — accurate, current, and citable, with far less hallucination and no retraining.
An LLM only knows what was in its training data — it's frozen, has a knowledge cutoff, can't see your private documents, and will confidently hallucinate when it doesn't know. RAG fixes this by retrieving relevant, current information at query time and giving it to the model as context, without any retraining.
First you prepare a knowledge base: load your documents, split them into chunks (passages sized to balance precision and context), embed each chunk into a vector, and store them in a vector database. This one-time pipeline turns your corpus into searchable semantic vectors.
At query time, embed the user's question and find the most similar chunks via nearest-neighbor search — often followed by reranking for precision. The top chunks are injected into the prompt as context, with instructions telling the model to answer from that context and admit when the answer isn't there.
The LLM generates an answer grounded in the retrieved passages, ideally citing sources. Because the facts come from your real, current data, answers are accurate, attributable, and updatable (re-index, don't retrain). The quality ceiling is set by retrieval: bad chunks in → bad answer out.
Key Concepts
Retrieval-Augmented Generation: fetch relevant context at query time and feed it to the LLM to ground its answer.
Splitting documents into passages sized for retrieval — too big loses precision, too small loses context.
Nearest-neighbor search returning the k most relevant chunks for the query embedding.
A second, more precise model that re-scores retrieved chunks to push the best ones to the top.
Instructing the model to answer only from provided context (and admit gaps), reducing hallucination.
Measuring retrieval quality (recall/precision) and answer faithfulness/relevance to detect regressions.
1// RAG at query time: retrieve relevant context, then generate grounded in it.2async function answer(question: string) {3 // 1) Embed the question and find the most similar chunks in the vector store4 const qVec = await embed(question);5 const chunks = await vectorDB.search(qVec, { topK: 4 }); // nearest neighbors67 // 2) (optional) Rerank for precision — a cross-encoder scores query↔chunk pairs8 const top = await rerank(question, chunks).then(r => r.slice(0, 3));910 // 3) Augment the prompt with retrieved context + explicit grounding instructions11 const prompt = `Answer using ONLY the context. If it's not there, say you don't know.1213Context:14${top.map(c => "- " + c.content).join("\n")}1516Question: ${question}`;1718 // 4) Generate — the model answers from the provided sources, and can cite them19 return llm.complete(prompt);20}2122// Offline (indexing) step, done once: load → CHUNK → embed → store in vector DB.
RAG is the most common pattern for production LLM applications — chatbots over docs, search assistants, copilots — because it delivers grounded, up-to-date, citable answers without the cost and staleness of fine-tuning. Understanding the retrieve-then-generate pipeline (and that retrieval quality is the bottleneck) is essential for AI engineering and a top interview topic.
Common Pitfalls
1A Support Bot That Knows Your Docs
A company wants an AI assistant that answers customer questions from its own help center, pricing, and policies — accurately and with citations — not generic guesses from the model's training data.
A bare LLM doesn't know the company's specifics, can't see internal docs, and hallucinates plausible-but-wrong policies. Fine-tuning on the docs is expensive, slow to update, and still prone to making things up.
Build a RAG pipeline: index the docs into a vector store, and at query time retrieve the most relevant passages and inject them into the prompt with grounding instructions. The bot answers from real content, cites sources, and stays current — just re-index when docs change.
Takeaway: RAG is the standard way to give an LLM private, current knowledge with citations and far less hallucination — without the cost and staleness of fine-tuning. Update knowledge by re-indexing, not retraining.
2Garbage Retrieval, Garbage Answer
A RAG system returns confident but wrong answers. Investigation shows it retrieves irrelevant chunks because documents were split mid-sentence and the top-k results don't actually contain the answer.
The team focused on the LLM and prompt, but the failure was upstream: poor chunking and weak retrieval fed the model irrelevant context, so even a great model produced bad answers. RAG quality is bounded by retrieval quality.
Fix the retrieval layer: chunk on semantic boundaries with sensible overlap, improve the embedding model, add a reranker to boost precision, and evaluate retrieval (does the answer-bearing chunk appear in top-k?). Better context in yields accurate, grounded answers out.
Takeaway: RAG lives and dies by retrieval. Invest in chunking, embeddings, reranking, and retrieval evaluation — the LLM can only be as good as the context you give it.
Join the discussion
Comments
Loading comments...
Cookie preferences
We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.