Getting a model to answer from your documents, without retraining it.
This course assumes you know what a token, an embedding and a context window are. If those words mean nothing to you, start with how an LLM works, then come back.
Imagine the situation. You have 300 PDFs: internal procedures, technical documentation, meeting notes. You'd like to ask a question in plain language, something like "what is the procedure if sensor X saturates?", and get an accurate answer from these documents, with the page it comes from.
You open ChatGPT. First problem: it doesn't know your documents, it learned from public text up to a certain date. Second problem, more insidious: it doesn't tell you when it doesn't know. A language model is trained to produce a plausible sequence of words, not to verify. Faced with a question it doesn't know the answer to, it makes one up that looks like a good answer. This is called a hallucination.
There are three possible answers. The first two don't work here, and seeing where they fail helps explain the third.
This is fine-tuning. You extend the model's training with your data. It sounds logical, and it's almost always the wrong tool here:
Since the model reads what you write to it, why not paste all 300 PDFs before the question? Because there is a context window: the amount of text a model can read at once. It is limited, and it is metered. 300 PDFs won't fit. And even if they did, you would pay for your entire documentation with every question, for accuracy that drops as useful information gets buried in noise.
The idea is simple once stated: you don't give everything, you give just what's needed. Before asking the model a question, you automatically search for the 3 to 5 relevant passages in the 300 PDFs, and paste them into the prompt along with the question. The model only has to compose an answer from what it sees.
It's an open-book exam.
Fine-tuning is memorizing the book (long, imprecise, to redo with each edition).
Pasting everything into the prompt is re-reading the entire library with every question.
RAG is opening the book to the right page, then answering from that.
Hence the name, RAG, for Retrieval-Augmented Generation:
The model does not search your document base. There is no access to it. It is your code that searches, finds the passages, and writes them into the prompt. The model only sees text that is slightly longer than usual. All the interesting work of RAG happens before the call to the model.
What this changes in practice: the answer becomes verifiable, since you know which document and page it came from. Updating knowledge takes a few seconds, the time to reindex a file. And the model stays generic, so you can swap it for another.
RAG in one image: two distinct phases
That leaves the hard part: how does a machine find the right passages? Searching for words from the question is not enough. If the document says "the sensor is in saturation" and you ask "what to do when X overflows?", no words match. You need to search by meaning, That is what embeddings are for, explained in the course on how an LLM works.
You now have the two building blocks. Let's assemble the machine. It works in two phases you must never confuse: one preparation done just once, and one loop replayed with every question.
Splitting and its overlap
You search on vectors, you answer with text. The vector finds, never writes. That is why you store both.
To compare whole passages, you don't want one vector per word: you want one single vector for an entire paragraph. The embedding model reads the whole passage, contextualizes each word, then averages. It outputs one point that sums up the whole thing.
The size of this vector is set by the model (384, 768, 1024, 3072 numbers), never by text length. A 5-word sentence and a 300-word paragraph both give a vector of the same size.
There are two distinct families of models: an embedding model (small and fast, it turns text into vectors for searching) and an LLM (large, it composes). They do different jobs and you never confuse them.
A RAG manipulates two different tokenizers. That's the trap that surprises everyone.
Two paths, two tokenizers
① Chunk by tokens, not characters → respect the embedding model's window.
② Count tokens (tiktoken) → don't exceed the LLM's context.
③ Remember the tax: a French corpus fills chunks faster, so you fit fewer.
It's the contextual embedding that matters, only it knows how to distinguish meanings. It is calculated once per chunk at indexing, then stored in your database.
By the angle between two vectors, called cosine similarity. The intuition: in this space, it is the direction that carries meaning, not the length of the arrow. Two texts pointing in the same direction talk about the same thing.
Small angle = same subject
You do not implement any of this. The vector database does the computing and optimization: you choose the metric (cosine) and the k, you call search(vector, k), you get back the passages sorted.
→ The full details (cosine formula, worked example, brute force vs HNSW, late chunking) are in cheat sheet on this page.
A RAG that answers poorly is almost always one of these six things.
| chunk size | 200 to 500 tokens. Too small: the passage loses context. Too large: the vector gets vague and finds poorly. |
| overlap | 10 to 15% of chunk size. Enough to not split an idea, not enough to duplicate the whole database. |
| k | 3 to 10 passages. Too few: information is missing. Too many: you overwhelm the model and pay tokens for nothing. |
| score threshold | Ignore chunks below ~0.8 cosine, rather than send off-topic stuff to the LLM "because you needed k results". |
| same model | The same embedding model for documents and questions, no exception. |
| count in tokens | Split by tokens, not characters, and remember that French text costs roughly 1.5× more. |
You store your chunks as vectors. For a question, you turn it into a vector and retrieve the vectors that are closest.
① Two models, not one: the embedding model (search) ≠ the LLM (answer).
② Window of the embedding model often 512 tokens → hence chunking by tokens.
③ Linguistic tax: a French chunk fills the window faster → fewer words per vector.
Vectors live in a vector database. Finding the closest among millions is done by approximate search (ANN).
RAG is not magic. Knowing where it breaks saves weeks of work.
RAG reduces hallucinations, it does not eliminate them. If the supplied passages are off-topic, the model will still write something. That is why showing sources matters: it makes the error visible.
Once you have a basic RAG in place, here is the order to improve it.
You now have the complete mental model: split, vectorize, store, find by proximity, paste into prompt, generate, cite.
Switch to the Sheets tab for the condensed version, with technical details, formulas, and reference diagrams.
From your raw documents to a sourced answer. Step by step.
Overview: two phases
The LLM does not search your database. It's you who searches, then you pass the found text to it in the prompt. It has no access to your database.
PDF, Word (.docx), web pages, source code... the knowledge you want to make queryable. At this stage, nothing is done yet: these are files on a disk.
An embedding model reads only text. We discard the formatting (fonts, columns, images), we keep the content.
① An entire document exceeds the window of the embedding model (often 512 tokens).
② A single vector for 50 pages would be too vague to find a specific piece of info.
Typical size: 200 to 500 tokens. We chunk by tokens, not by characters.
Chunks + overlap
The chunk's vector is its address in the meaning space. Two chunks with similar meaning = nearby vectors, even if they don't use the same words.
The internal mechanism
Attention only links tokens within the chunk itself. Each chunk is encoded in isolation: the vector of chunk 2 knows nothing about chunk 1 or 3.
Attention does not cross boundaries
Late chunking: reverse the order, embed the entire document first (each token sees its neighbors), and only split the vectors afterwards, just before pooling. Each chunk vector has therefore "seen" its neighbors. Cost: a long-context model and more computation.
Contextual retrieval: prepend a short document summary to the head of each chunk before encoding it.
Sentence-window: encode small chunks, but return to the LLM the chunk plus its neighbors when answering.
Isolated chunking remains the default: simpler, and a vector overloaded with context retrieves less sharply.
① the vector (the search key) · ② the original text (this is what we'll send to the LLM) · ③ the metadata (source, page → to cite).
What the database contains
| ID | Vector (key) | Chunk text | Source |
|---|---|---|---|
| 1 | [0.07, 0.5, …] | "Low temperature cooking..." | guide.pdf · p.3 |
| 2 | [−0.2, 0.1, …] | "Preheat the oven..." | guide.pdf · p.4 |
| 3 | [0.4, −0.3, …] | "def cook(temp): …" | script.py |
The vector serves to find. The text serves to answer. This entire phase (§1 to §5) is done only once.
The question is encoded by exactly the same embedding model as the chunks. Otherwise the vectors live in different spaces and are not comparable.
The array gives the coordinates of a single point. Each cell = one axis. 768 numbers = 1 point in a 768-dimensional space, not 768 points.
The array = the coordinates of a point
Meaning is carried by the direction. So to compare two vectors, we measure the angle between them.
Small angle = nearby meaning
cos(θ) = (A·B) / (‖A‖ × ‖B‖)
The dot product divided by the lengths, which cancels the size effect: only direction matters.
If the vectors are normalized (length 1): cosine = simple dot product. That's why it's fast.
1. Dot product: A·B = a₁b₁ + a₂b₂ + … + aₙbₙ (multiply term by term, sum).
2. Length (Pythagoras): ‖A‖ = √(a₁² + a₂² + … + aₙ²).
3. Divide. (If you want the angle: θ = arccos(cos θ), but for search we stop at cosine, which is the score.)
Example. A = [2, 1], B = [1, 3]
A·B = 2×1 + 1×3 = 5
‖A‖ = √5 ≈ 2.24 · ‖B‖ = √10 ≈ 3.16
cos θ = 5 / (2.24 × 3.16) ≈ 0.71 → about 45 degrees.
0.71 means "fairly close". In practice we mainly keep scores > ~0.8.
The cosine of an angle = the horizontal coordinate of the point on the circle. The smaller the angle, the further right we are, the closer the cosine is to 1.
For RAG: we aim for cos ≈ 1 → angle ≈ 0 → same direction.
Brute force = calculate the cosine with every vector, sort, keep the top-k. Exact, but N calculations per question.
ANN (approximate) = an index compares only a small fraction. Tiny chance of missing the closest one.
Brute force vs HNSW
It's the vector database that does everything. You choose the metric (cosine) and the k, you call search(vector, k), you get back the k chunks sorted. You don't implement cosine or HNSW.
A prompt in 3 parts: instruction + context (the text of the k chunks) + question.
Prompt assembly
The answer is grounded in specific, identified passages → fewer hallucinations. The model has the material right in front of it instead of answering from memory.
Thanks to the metadata kept in storage (§5), we display the sources: which documents, which pages helped. The answer becomes verifiable.
Only steps §6 to §10 are replayed with each question.
Indexing (§1 to §5) is redone only if your documents change.