John Carioto · October 11, 2026
The question
When you type a question into ChatGPT or Claude, nothing goes off and looks up the answer. The model predicts one word, adds that word to what it has read so far, and predicts the next one, until it predicts that it's done. That much gets repeated everywhere. What I wanted to see was what "predicts" means physically: which numbers get multiplied by which, and how a capital city falls out the other end.
The answer is arithmetic, and a lot of it. The rule of thumb for a model the size of GPT-3 is roughly 175 billion multiply-and-adds for every word it writes. Georgia Tech's Transformer Explainer is a great visualization of this. It runs the real 124-million-parameter GPT-2 in your browser, and the visuals are excellent. But GPT-2 carries 768 numbers for every word, so the figure has to show them as colored bars. You can watch the data flow, but you can't check the math.
The model on this site is the opposite trade: a real transformer with the same parts as GPT-2, shrunk until every number fits on a screen, and trained to answer simple questions. The interactive explainer lets you ask it something and click through every step. This article walks through one question, "what is the capital of france ?", with the real numbers.
A model small enough to check by hand
| This model | |
|---|---|
| Vocabulary | 107 words |
| Numbers per word (vector size) | 16 |
| Blocks | 2 |
| Attention heads per block | 2 |
| MLP neurons per block | 64 |
| Context window | 16 tokens |
| Trained parameters | 9,984 |
| Training data | 185 short questions and answers |
The training set covers five kinds of question: capitals ("what is the capital of france ?"), colors ("what color is the sky ?"), animal sounds, small sums ("what is two plus three ?") and opposites. Each fact was asked several ways, and one wording of every third fact was held back. The model never saw those 24 questions and answers all 24 correctly, so it learned to find the fact in the question rather than memorize exact sentences. Training took about two minutes on an ordinary CPU.
Every answer has the same shape, "it is paris" or "it says moo", followed by an end marker. Training used label smoothing, which tells the model that the right word is about 90% likely rather than certain. That's why it reports "paris" at 89% instead of 100%, and it keeps the numbers in a readable range.
Step 1: words become numbers
The model can't read letters. Each word in the vocabulary has an ID, and a special start token, <bos>, goes in front:
<bos> what is the capital of france ?
0 2 3 4 5 6 24 7
Each ID picks one row of the embedding table, E, which has 107 rows of 16 numbers learned in training. The row for "france" starts like this:
E[france] = [-0.100, 0.060, -0.390, -0.025, -0.342, 0.220, 0.255, -0.476, ...]
A second table adds a learned vector for each position, so the model can tell where in the sentence a word sits. The sum is the word's starting vector. Real models do the same thing with pieces of words instead of whole words, which is how they can read a word they've never seen, and with vectors of 768 numbers (GPT-2) to more than 12,000 (GPT-3) instead of 16.
The residual stream
Here is the whole pass, drawn as a signal-flow diagram:
Click the diagram to open it full size. The explainer has the live version, where every box opens its math.
The thick line is the residual stream. Each word's 16 numbers travel along it. Each block has two halves that tap off the line, compute something, and add the result back at a summing junction. Nothing on the main line is ever replaced, only added to. The thin strips on the line are the last word's vector at each point; the triangles in the attention boxes are the attention patterns.
Each half starts with a LayerNorm, which rescales the vector to an average of 0 and a standard deviation of 1. That keeps the numbers in a steady range no matter how much has been added upstream.
Step 2: attention, where words look at each other
Attention is the only place where one word's numbers can affect another's. Each word's normalized vector is multiplied by three learned 16 × 16 matrices to make three new vectors:
- a query: what this word is looking for
- a key: what this word offers the others
- a value: what this word hands over if it's chosen
The 16 numbers are split into two heads of 8, and each head runs attention on its own. For every pair of words, the head takes the dot product of one word's query with the other's key (multiply the 8 matching numbers and add them up) and divides by √8. A word can only look at itself and the words before it. Later words are masked out, because when the model is writing they don't exist yet. Softmax then turns each row of scores into weights that are positive and add up to 1:
score(i, j) = (q_i · k_j) / √8
weight(i, j) = e^score(i, j) / Σ_k e^score(i, k)
output_i = Σ_j weight(i, j) × v_j
On the pass that writes the answer, the input is "what is the capital of france ? it is", and the last word, "is", is the one making the prediction. In block 1, head 1, its query against the key of "france" gives a dot product of 9.889. Divided by √8, that's a score of 3.496, well above every other word in the row. After softmax, "france" gets 94.6% of that head's attention, and the head's output is almost entirely the value vector of "france". The model has fetched the country.
The cleanest example is arithmetic. Ask "what is two plus three ?" and, on the answer pass, the two heads of block 1 split the job:
Head 1 puts 99.3% of its attention on "two" and head 2 puts 99.8% on "three". Nobody told the heads to do that. The division of labor came out of training because it was the easiest way to get the sums right.
Not every head is that tidy. On the France question, head 2 of block 1 puts 48% of its attention on the <bos> start token, which is a common way for a head to do very little. Large models show the same habit, and researchers call it an attention sink.
Step 3: the MLP
After attention mixes information between words, each word goes through the MLP on its own. The vector is normalized again, multiplied out to 64 numbers (the neurons), passed through a function called GELU, and multiplied back down to 16:
u = h · W1 16 → 64
g = GELU(u) keeps positive values, squashes negative ones toward 0
out = g · W2 64 → 16
x = x + out back onto the residual stream
GELU is a smooth version of "if it's negative, make it zero." That bend matters. Without it, the MLP's two matrix multiplications would collapse into a single matrix, and the 64 neurons would add nothing one matrix couldn't do.
In large models, the MLP layers appear to be where much of the memorized knowledge lives; one well-known paper describes them as key-value memories. Here, attention brings "france" to the last position, and the layers after it turn that into a vector that scores highest against "paris".
Step 4: scoring every word
After the second block, only the last word's vector matters for the prediction. It gets one more LayerNorm and is then compared, by dot product, with a learned row for each of the 107 words in the vocabulary. Those 107 numbers are the logits, and softmax turns them into probabilities:
"paris" scores 6.31 and the runner-up, "beijing", scores 0.50. The softmax arithmetic: subtract the top score from every logit, raise e to each result, and add them up. The total is 1.120. "paris" contributes e⁰ = 1, so its probability is 1 / 1.120 = 89.3%. The rest is spread thinly over the other 106 words, which is the label smoothing showing through.
This model always takes the top word. Chatbots usually don't: they roll dice weighted by these probabilities, and a setting called temperature divides the logits first to make the odds sharper or flatter. That's why the same question can get different answers from the same model.
Step 5: one word at a time
That was one pass. The model now appends "paris" to the input and runs the entire calculation again, from the table lookup on, to get the next word. The full answer took four passes:
| Pass | Input ends with | Prediction |
|---|---|---|
| 1 | … capital of france ? | it (90.1%) |
| 2 | … of france ? it | is (90.1%) |
| 3 | … france ? it is | paris (89.3%) |
| 4 | … ? it is paris | <end> (90.1%) |
The pass that wrote "paris" read 10 tokens and took about 67,000 multiplications. The whole answer took about 253,000. GPT-3 would spend roughly 175 billion on each word. Production systems also keep each word's keys and values between passes, the "KV cache," so each new pass only computes the newest word. The result is the same; it just skips repeated work.
Where it breaks
The useful thing about a model this small is that you can watch it fail.
It generalizes when a question is close to what it trained on. "capital of france ?", a wording it never saw, still gets "paris" at 91.5%, because head 1 of block 1 finds "france" (94.9%) wherever it sits in the sentence.
Ask something outside its patterns and the arithmetic still produces an answer. "what sound does a banana make ?" gets "it says yellow" at 30%. The attention heads find "banana" just as they found "france", and the only thing the model ever learned about bananas is their color. The low probability is a hint that something is off, but it isn't a reliable alarm: "what is the opposite of france ?" gets "paris" at 80%.
Words outside the vocabulary are refused outright. A real model would split an unfamiliar word into pieces it does know and carry on, which is more useful, and also how large models can sound fluent about things they've never seen.
None of this is a bug in the arithmetic. In engineering terms, the model is a curve fit with 9,984 coefficients. Like any curve fit, it interpolates well, extrapolates badly, and has nothing built in to tell you which one you asked for. Large models are far better at it because their training covers far more ground, but the mechanism is the same.
Same math, bigger numbers
| This model | GPT-2 small (2019) | GPT-3 (2020) | |
|---|---|---|---|
| Trained parameters | 9,984 | 124 million | 175 billion |
| Numbers per word | 16 | 768 | 12,288 |
| Blocks | 2 | 12 | 96 |
| Attention heads per block | 2 | 12 | 96 |
| MLP neurons per block | 64 | 3,072 | 49,152 |
| Vocabulary | 107 words | 50,257 tokens | 50,257 tokens |
| Context window | 16 tokens | 1,024 | 2,048 |
| Training text | 185 short Q&As | about 40 GB of web pages | about 300 billion tokens |
Current models add refinements this one leaves out: rotary position encoding instead of a position table, RMSNorm instead of LayerNorm, gated MLPs, attention heads that share keys and values, and mixture-of-experts layers that run only part of the network for each word. They're trained on trillions of words, then trained further to follow instructions. The operations underneath are the same lookup, dot product, softmax and add.
The interactive explainer runs this exact model in your browser. Ask it something, then click a number to see where it came from.
Further reading
- Vaswani et al., "Attention Is All You Need" (2017). The paper that introduced the transformer.
- Radford et al., "Language Models are Unsupervised Multitask Learners" (2019). GPT-2, the layout this model follows.
- Brown et al., "Language Models are Few-Shot Learners" (2020). GPT-3.
- Cho et al., "Transformer Explainer: Interactive Learning of Text-Generative Models" (2024), and the live demo.
- Geva et al., "Transformer Feed-Forward Layers Are Key-Value Memories" (2021).
- Xiao et al., "Efficient Streaming Language Models with Attention Sinks" (2023).
- Jay Alammar, "The Illustrated Transformer." A friendly visual walk-through.