From raw text to a written answer: the three mechanisms behind a language model. No prior knowledge needed.
A language model is a program that has learned, from enormous amounts of text, to answer a single question: what is the most likely piece of text after these ones? Everything else follows from that. It queries no database and searches nowhere: it computes a probability.
Through four mechanisms, which are the next four chapters. The text is first cut into numbered pieces, that is the tokenizer. Each number then becomes a position in a space where meaning can be computed, that is the embedding. Those positions correct one another according to the sentence, that is attention. Finally the model produces its answer one piece at a time, that is generation.
The full journey
A model's parameters, also called its weights, are the numbers adjusted during training. They run into the billions. They carry everything the model retained, and they do not change afterwards unless the model is retrained.
Seven words are enough to read everything else. Come back to this if a term blocks you.
| token | A piece of a word. The smallest unit the model handles. "Hello" might be 1 token, "Schtroumpfology" 4. |
| weights | The numbers learned during training, also called parameters. An "8B" model holds 8 billion of them. They stop moving once training is over. |
| attention | The mechanism that lets each token look at the others and correct its vector accordingly. It is the heart of the Transformer architecture, the one behind every current LLM. |
| vector | A list of numbers, for example [0.2, −0.7, 0.1, …]. This is what the machine actually computes. |
| embedding | The vector that represents a text. Its special property: two texts with similar meaning have nearby vectors. |
| context window | The maximum amount of text a model can read at once, counted in tokens. |
| LLM | Large Language Model, the model that composes the final answer (GPT, Claude, Mistral…). |
A computer only handles numbers. Before anything else, you have to turn "cat" into something computable. This is the job of the tokenizer, the entry point to every language model.
Three splits are possible, and you understand the right one by ruling out the other two:
The path of text, toward meaning
Each piece gets a number (its ID), looked up in a fixed dictionary. And this is where you need to be very clear about one thing:
These numbers mean nothing. They are arbitrary labels, like barcodes: IDs 331 and 332 have no relation to each other. The tokenizer understands nothing. It counts, it splits, it numbers.
The number serves only as an address: it tells which row to read in the big table where vectors are stored, the embedding table. The dictionary (text → number) is made once then fixed; the table (number → vector) is learned during training and is part of the model.
From a statistical algorithm called BPE (Byte Pair Encoding), which is not intelligent. You start with characters, then repeat the same operation thousands of times: merge the most frequent pair of symbols in the corpus. "e" followed by "s" appears everywhere, so you create "es". Then "es" + "t" becomes "est". After roughly 100,000 merges, you lock the result: that's your vocabulary.
BPE: merge what appears often
Two important consequences follow from this mechanism.
First consequence: nothing can break. Under BPE merges, you always keep the 256 basic bytes. An unknown emoji, an exotic character, a corrupted file? The tokenizer falls back to the raw byte, one token per byte. It fragments, but never stops. This is called byte-level BPE.
Second consequence: the vocabulary is optimized for English, because it was mostly extracted from English. Other languages are therefore split more finely, so more tokens to say the same thing:
Tokens needed for 100 words
1. Everything is counted in tokens, never in characters or words, since model limits are expressed that way.
2. A tokenizer is married to its model: its numbers point to the right vectors only for that model.
→ The full details (merges by rank, KV cache, O(N²) cost, tiktoken) are in Sheet 01, under "Sheets".
At this point, you only have arbitrary numbers. Meaning arrives at the next step, with the idea that makes everything else work.
The answer: you make it a position in space. Every word, every passage, becomes a point. And you arrange things so that points with nearby meaning are physically close.
The space of meaning (simplified 2D view)
That is what an embedding is: a position. And that is what lets the model treat two different phrasings alike. If "the sensor saturates" and "the sensor overflows" land at the same place, it understands them the same way, without them sharing a single word.
They are learned, on a simple principle: a word is defined by its surroundings. The model reads billions of sentences, and two words you see in the same patterns ("feed the ___", "the ___ sleeps") are pushed toward nearby positions. No one declared that cats and dogs are alike: it is inferred from usage.
Each model learns its own space. One model's coordinates mean nothing to another. You never mix vectors from two different models. Two models do not speak the same geometric language, and nothing flags the mistake.
"Bank" means the edge of a river and also a place that holds money. If each word has only one fixed position, where do you put it? Answer: in the middle, so in neither zone, which helps no one.
One point for two meanings: the compromise
The solution is called attention: before locking a word's position, the model lets it look at its neighbours in the sentence and shifts its position accordingly. "We fished from the bank" sends it towards rivers, "the bank approved my loan" towards finance.
It is the central mechanism of Transformers, and it deserves more than one sentence. It is the whole of the next chapter.
→ The arithmetic of directions (king − man + woman ≈ queen) and the detail of static vectors are in Card 02, "Cards" tab.
The previous chapter ended on a problem: "bank" has only one position, stuck between rivers and money. Attention is the mechanism that moves it. It is the central piece of Transformers, and the only place in the whole model where tokens talk to each other.
To keep this concrete, this chapter and the next follow a real model whose numbers are public: Llama 3 8B. Every figure quoted is its own.
Once the tokenizer and the embedding table have done their job, your text no longer exists as text. It is a table: one row per token, and on each row 4,096 numbers.
"the black cat sleeps" entering the model
Position information is then added. Without it, "the cat eats" and "eats the cat" would be strictly identical to the model: nothing in a table of numbers says in which order to read the rows.
Two methods exist. Older models (GPT-2, BERT) add a second vector to the input vector, one that encodes "I am at position 1, 2, 3…".
Recent models, Llama among them, use RoPE: instead of adding anything, they rotate the Q and K vectors by an angle proportional to the position. Useful consequence: the score between two tokens then depends only on the gap between their positions, not on their absolute positions. That is what makes it possible to extend a model's context window after training.
Each token asks the others a question, each one answers with a label, and the asker mostly picks up the content of those whose label matches. Three roles, so three vectors, all drawn from the same starting vector.
Three vectors drawn from the same input vector
A matrix is a grid of numbers. The multiplication produces a new vector in which each output number is a weighted sum of all the input numbers. Nothing more.
With a 3 number vector and a 3 × 2 matrix:
| Input | Weights toward output A | Weights toward output B |
|---|---|---|
| 2 | 0.5 | −1.0 |
| 5 | 0.2 | 0.3 |
| 1 | 1.0 | 0.4 |
| Output | 2×0.5 + 5×0.2 + 1×1.0 = 3.0 | 2×(−1.0) + 5×0.3 + 1×0.4 = −0.1 |
Inside the model it is identical, with 4,096 inputs and 4,096 outputs. That takes 4,096 × 4,096 = 16.8 million weights, which is exactly the size of WQ in the next chapter's table.
Multiply term by term, add them up, and out comes a single number.
Q = [2, 5, 1] and K = [1, 0, 3] give 2×1 + 5×0 + 1×3 = 5.
That number measures how well the two vectors line up:
The scores are then divided by the square root of the vector size, here √128 ≈ 11.3. Without that division, adding 128 products gives huge values, the softmax locks onto a single candidate, and training never gets going.
A softmax turns any list of numbers into probabilities. Two operations: apply the exponential to each score, then divide each one by the total.
The exponential makes everything positive, crushes small gaps downward and stretches large ones upward. The division guarantees a total of 1.
| Raw score | Exponential | Divided by the total |
|---|---|---|
| 3.0 | 20.1 | 20.1 / 48.4 = 41% |
| 2.4 | 11.0 | 11.0 / 48.4 = 23% |
| 1.9 | 6.7 | 6.7 / 48.4 = 14% |
| 1.2 | 3.3 | 3.3 / 48.4 = 7% |
| all the others | 7.3 | 7.3 / 48.4 = 15% |
| Total | 48.4 | 100% |
The same computation is used twice in the model: here to spread attention over past tokens, and at the very end to spread probability over the 128,256 tokens of the vocabulary.
Who looks at whom, and how much
The grey zone is not a detail. A token never sees what follows it, which forces the model to write left to right without cheating, and makes the cache described in the next chapter possible.
Steps 2 and 4 are the only moment in the whole model where tokens communicate. And they use no weights at all: they are nothing but products between vectors. The weights sit in the three projections of step 1, and in the final blend.
A single set of Q, K, V could only track one relationship at a time. So it gets cut up: the Q, K and V vectors are sliced into 32 slices of 128 numbers, and the four steps run separately on each slice.
The Q vector, sliced into 32 heads
Each head specialises during training: one tracks the subject of the verb, another gender agreement, another the quotation mark still to be closed. The 32 results are stitched back together, which rebuilds 4,096 numbers, then blended one last time by WO. That fourth matrix is what decides how to combine whatever the 32 heads brought back.
In the next chapter's table, WQ is 4,096 × 4,096 while WK and WV are 4,096 × 1,024. That is not a typo.
Llama 3 uses grouped-query attention: the 32 query heads share only 8 sets of K and V, in groups of 4 heads. Quality barely moves, and the cache memory is divided by 4.
Take "we fished from the bank". The token "bank" emits a Q which, thanks to training, lines up well with the K of "fished". That neighbour therefore collects a high attention weight, and its V weighs heavily in the blend. Result: the vector for "bank" shifts towards the river zone. In "the bank approved my loan", it is the K of "loan" that wins, and the vector moves towards finance.
The same word, two output positions
The vector that enters attention is the static embedding, read from the table. The one that comes out is the contextual embedding. And since the model stacks 32 layers, attention runs 32 times: from the second one onward, it is already working on contextual vectors.
→ The static versus contextual distinction, and the two moments where context acts, are covered again in Card 02, "Cards" tab.
Attention is understood. What remains is where it sits inside the complete machine, and how a table of numbers becomes a written word.
The model produces one token at a time. For each token produced, every single one of its weights is used, and all the text already there is traversed again.
The weights are not a formless heap. They are matrices, held in 32 layers stacked on top of one another. Each layer contains exactly seven matrices: the four attention ones you have just met, and three others. The count is checkable:
Full anatomy of Llama 3 8B
| Block | Matrix | Size | Weights |
|---|---|---|---|
| Attention | WQ | 4,096 × 4,096 | 16.8 M |
| WK | 4,096 × 1,024 | 4.2 M | |
| WV | 4,096 × 1,024 | 4.2 M | |
| WO | 4,096 × 4,096 | 16.8 M | |
| MLP | gate | 4,096 × 14,336 | 58.7 M |
| up | 4,096 × 14,336 | 58.7 M | |
| down | 14,336 × 4,096 | 58.7 M | |
| One layer | 218 M | ||
| × 32 layers | 6.98 B | ||
| Embedding table + output layer | 1.05 B | ||
| Total | 8.03 B | ||
① The 32 layers share the same structure, but no two hold the same values. The model repeats the same operation 32 times with 32 different sets of weights.
② 4 weights out of 5 sit in the MLP, not in attention. Attention is the famous mechanism, but it is not where the bulk is stored.
Always the same two, in this order.
The contents of one layer
The second move is called the MLP, short for multi-layer perceptron. Once attention is done, every row of the table goes its own way: its 4,096 number vector is stretched to 14,336, run through a non-linear function, then brought back to 4,096. No exchange between tokens here: the same operation is applied to each row, separately.
This is where 4 weights out of 5 live. The fairest picture: attention gathers the useful information, the MLP processes it. Reverse engineering work on models locates most of the learned factual associations here.
Without it, stacking 32 layers would achieve strictly nothing. A chain of matrix multiplications always collapses into one equivalent matrix: 32 linear layers would have exactly the same power as a single one.
The non-linear function breaks that collapse. It is what makes depth useful. In Llama it is a variant called SwiGLU, and it is the reason the MLP has three matrices instead of two: gate and up expand in parallel, the first acting as a valve on the second.
Before each of the two moves, the vector passes through a normalisation that brings its numbers back to a stable scale (RMSNorm in Llama). Without it, values blow up or collapse across the 32 layers and training fails.
It changes nothing in the reasoning above, but if you open a Transformer diagram you will see those little "Norm" blocks everywhere, and now you know what they do.
Neither of the two moves returns a fresh vector: its result is added to the vector that came in. A token's vector therefore crosses the 32 layers being enriched 64 times, without ever losing what it carried at the start.
That shortcut has a name, the residual connection. Without it a 32 layer stack would be close to untrainable: the signal degrades on the way down and the early layers stop learning anything.
Crossing the stack
Coming out of the 32nd layer, only one row of the table is kept: the last token's. The others served to feed it, and are of no further use at that instant.
That 4,096 number vector is multiplied by one final matrix of 4,096 × 128,256. Out come 128,256 scores, one per vocabulary entry. Their technical name is logits, and you will meet it everywhere in the documentation. Each score is the dot product between the final vector and that token's output vector: the better aligned, the more likely.
Those raw scores are not probabilities: they can be negative, and there is no reason for them to sum to anything in particular. The softmax turns them into positive probabilities that add up to exactly 1. It is the same operation as in the previous chapter, applied this time to the whole vocabulary.
"The cat is sleeping on the ___"
Always taking the most likely token gives correct but flat text, and sometimes loops of repetition. So a controlled amount of randomness is added, tuned by temperature.
Its mechanism fits in one line: the raw scores are divided by the temperature before the softmax. Dividing by 0.5 doubles the gaps, so it concentrates probability on the favourites. Dividing by 1.5 shrinks them, so it flattens the distribution and lets rare choices through.
Same raw scores, three temperatures
T = 0.5 · predictable
T = 1.0 · the model's raw output
T = 1.5 · unpredictable
A second setting, top-p, works differently: it sorts candidates by decreasing probability and keeps only the first ones until it reaches p, say 0.9. Everything else is discarded before the draw, which stops an absurd candidate from slipping out by bad luck.
The loop, token after token
Every weight in the model is used at least once, which is roughly 16 billion operations (one multiplication and one addition per weight). At 50 tokens per second, the machine is sustaining 800 billion operations per second for your answer alone. That is why this runs on GPUs.
What precedes describes a dense model: every weight serves every token. Llama 3 8B is one.
Other models use a mixture of experts: the MLP is split into blocks, and a router activates only a few of them per token. DeepSeek-V3 advertises 671 billion parameters, but puts only 37 billion to work for each token produced. A large model in memory, a small model in compute.
"It rereads everything on each pass" is true mathematically, but false inside the machine. The K and V of tokens already processed never change: they depend only on the token and its position, never on what comes after. So they are kept in memory.
The first pass therefore takes the whole prompt through the 32 layers at once. After that, on every turn, a single token crosses the stack: it computes its Q, and reads everyone else's K and V straight from the cache.
Memory, and it grows with every token. On Llama 3 8B: 32 layers × 8 heads × 128 numbers × 2 (K and V) = 65,536 numbers per token, so 128 KB. A full 8,192 token window is 1 GB of GPU memory, on top of the model's own 16 GB (8 billion weights at 2 bytes each). It is often the cache, not the model, that caps how many users can be served in parallel.
Everything the model has in front of it at a given instant, your question, whatever you paste along with it, and its own answer in progress, has to fit inside its context window, measured in tokens. Past that, something has to be cut.
Two costs grow with length, but not at the same rate. The projections and the MLP cost in proportion to the number of tokens. Attention compares each token to all the preceding ones, so its cost grows with the square of the length. On a short prompt the first one dominates; on a very long prompt the second ends up drowning everything else.
Nothing in this mechanism verifies anything. The model optimises the plausibility of the next token given what it read during training, not truth. Faced with a question whose answer is not in its weights, the most plausible continuation is still a well formed sentence, so it produces one. That is a hallucination: a consequence of how it works, not an occasional breakdown.
There are two ways to reduce it: through training, by teaching the model to recognise when it does not know, and above all by handing it the material directly in its prompt.
→ That second method has a name and a course of its own: RAG, giving your own documents to the model.
Text → sequence of integers. Nothing else. Meaning comes later.
The complete journey, from text to meaning
The tokenizer never touches meaning. An ID is an arbitrary label, like a barcode: 331 and 332 have no connection. Meaning is learned much later, in the embedding table.
The model never reads text. It reads vectors. The ID just fetches the right vector from memory by simple index access.
The output is an array of integers. Thanks to BPE (§3), a frequent word fits in a single ID.
Frequent = 1 token · rare = many
The space before a word is stuck to the token. So chat and chat can give different IDs depending on position in the sentence.
Two tables. The dictionary (text → ID) is frozen, delivered in a few MB. The embedding table (ID → vector) is learned and is part of the model.
The ID is just a row number
| token | ID |
|---|---|
| Good | 12 |
| day | 8973 |
| ch | 331 |
| at | 266 |
| 🦊 | → bytes |
| row | learned vector |
|---|---|
| 12 | −0.1 0.4 0.8 |
| 266 | 0.2 −0.3 0.1 |
| 331 | 0.9 0.3 −0.5 |
| 8973 | 0.0 0.6 −0.2 |
That's what ties a tokenizer to its model: the same IDs point to the right vectors only for the model trained with that specific vocabulary.
Two stacked layers. Below: the 256 bytes → total coverage, nothing can crash. Above: BPE → efficiency, a known word = 1 token.
The stack
The dictionary is built by repeatedly merging the most frequent pair:
Building by merging
The merged pair is most frequent across the entire corpus, not in this isolated word. "es" appears everywhere, "ow" almost never.
Compilation (outside AI): a statistical program repeats merges until a fixed size (e.g. 100,000), then locks.
Inference (production): incoming text is re-split by replaying these merges in their learning order, which amounts to reconstructing the largest known blocks.
It's the merge order that decides, not the "longest prefix". That's more WordPiece (BERT). BPE applies its rules by rank.
The unknown word (Out Of Vocabulary) no longer exists. At worst, you fall back to raw bytes: 1 token per byte.
Fallback to bytes
Extreme splitting = fragmentation. Two effects on the machine: the O(N²) cost of attention, and KV cache saturation (VRAM).
O(N²) over the total sequence length.
A word going from 1 to 10 tokens in a 2,000-token context adds only 9 tokens, not a 100× factor. The 100× only applies if that word was the entire input. But cumulative across a whole text, fragmentation adds up fast.
The BPE vocabulary is over-optimized for English. Everything else costs more tokens, so more expensive and slower.
Tokens per 100 words
Consequences: more latency (time-to-first-token), more memory bandwidth, higher API bill.
Not "one tokenizer per company" but one tokenizer per model lineage. You don't plug one model's tokenizer into another: the IDs would point to wrong rows.
One encoding shared by a lineage
Yes, almost always, including at OpenAI. pip install tiktoken and you see exactly how text is split, locally. What's closed are the weights. The tokenizer isn't the model.
The ID becomes a position in space. That's where meaning appears.
The semantic space (2D projection of ~768 dimensions)
An embedding is a position, not a label. The ID 331 meant nothing. Here it's the opposite: the place in space IS the meaning. Two close vectors = two close meanings.
The embedding is simply the contents of the row pointed to by the ID. Mechanically trivial. The whole leap is in what these numbers mean.
① Proximity → similarity of meaning.
② Direction → type of relation.
Directions encode relations
A word is defined by its surroundings. Two words appearing in similar contexts are pushed toward similar vectors.
"cat" and "dog" come closer because you encounter them in the same patterns ("feed the ___", "the ___ sleeps"). Meaning is never declared: it is deduced from co-occurrences.
The space depends on the model that learned it. Two models → two unrelated spaces. Same lock as the tokenizer (sheet 01 §7).
There is no one embedding, but two. They don't serve the same purpose.
Attention is what separates the two meanings
Type 1 = the input to the model, right after the tokenizer.
Type 2 = the output, after the attention layers.
Context acts at TWO different moments
The pipeline in order: text → tokenizer (IDs) → static vectors → attention → contextual vectors.
Where does the STATIC vector of a two-sense word land?