machine learning24 min read
Exploring Transformer Architectures
An interactive exploration of attention, recurrence, and compositional computation through toy models and published research.
Contents
If you have written software for a while, you carry a lot of intuition about how programs are organised. You know when to write a function and when to inline it, what breaks when two services disagree about a data format, and why a loop is not the same as pasting a block of code ten times.
When I started looking closely at transformers, I kept reaching for those intuitions. Some transferred beautifully; others broke in instructive ways. This essay follows one question through a handful of published ideas: what kind of computer is a transformer, and what would it take to make it compute differently?
I assume you know GPT-2 (1), the decoder-only descendant of the original transformer (2): token embeddings, a stack of blocks that each contain attention and a feed-forward network (MLP), residual connections, and a final projection back to the vocabulary. Everything else is introduced along the way.
One note before we start. Every interactive figure here is a toy model or an exact calculation that runs in your browser. None of them is a trained neural network, and none shows a recorded experimental measurement. They are there to build intuition; the papers I cite provide the evidence.
A transformer is a program that writes to shared memory
Take one position in a sequence, say the token born in “Where was the author of The Glass Orchard born?”. GPT-2 keeps a vector of width for it, and that vector travels upward through the network. Each block reads it, computes something, and adds its result back:
x = embed(tokens) # (T, d): one vector per position
for block in blocks: # L blocks, own weights each
x = x + block.attn(norm(x)) # communicate across positions
x = x + block.mlp(norm(x)) # compute within each position
logits = unembed(norm(x))
Elhage and colleagues call this vector the residual stream and suggest thinking of it as a shared channel that every layer reads from and writes to (3). To a software engineer it looks like a shared blackboard. Attention is the only operation that moves information between positions, a bit like message passing between workers. The MLP transforms each position in place, like a pure function applied to each worker’s local state.
This picture invites a lens I will use throughout: composition of functions. If one layer computes , a later one and a later one , the network computes something like
where ‘s output becomes ‘s input, and ‘s output becomes ‘s input. Two consequences of this humble equation will follow us through the essay. First, can only use what produced if comes after , and a transformer has a fixed number of layers to come after. Second, has to read ‘s output the way meant it, and in a trained network nobody ever wrote that format down.
The first consequence is the natural place to start: how many steps does a transformer actually get?
Depth is a budget
Some computations come with long chains of dependencies. In the straightforward way to follow a chain of pointers times, or to carry digits across a long addition, step needs the result of step . Cleverer parallel algorithms exist for both (pointer jumping, carry-lookahead adders), but they need more total work or extra structure, and a network has to learn them. GPT-2 gives every token exactly blocks per forward pass, each with its own weights. So the number of layers fixes two different things at once: how much the model can store (parameters) and how many dependent steps it can execute (sequential depth).
A looped, or recurrent-depth, transformer pulls these apart. It keeps distinct blocks and applies the whole group times with the same weights, exactly like a loop body:
for r in range(R): # R iterations: more depth
for block in blocks: # only B distinct weight sets
x = block(x) # attn + mlp with their residual adds, as above
This is an old and fruitful idea. The Universal Transformer applied one shared block repeatedly, with an optional learned stopping rule (4). ALBERT shared one set of parameters across all of BERT’s layers to shrink the model (5). Looped transformers turn out to be expressive enough to emulate simple programmable computers (6), and recurrent-depth language models can spend more iterations on harder inputs at test time (7). The appeal is clear: if knowledge and sequential steps are separate resources, perhaps you can buy more of one without paying for the other.
Notice what the counters do as you add iterations. Parameters stay at blocks, while sequential depth and compute grow to blocks. Weight sharing saves parameter memory, not arithmetic: at the same number of block executions, each iteration still performs a full block’s worth of matrix multiplications, and activations and the attention key/value cache still grow with every executed block. That makes “is looping better?” an incomplete question. Compared with a plain model of equal parameters, the looped one gets roughly times more block compute, an unfair advantage. Compared with a plain model of equal compute, it has roughly times fewer block weights to hold knowledge in. These ratios apply to the transformer blocks; the embeddings and output layer are the same in both. We will need to come back to this when we talk about evaluation.
Loops raise a subtler problem too. Each iteration rewrites the state, so after a few passes the network may have lost track of the original question, like a loop body that receives only the running accumulator and not the original arguments. Bansal and colleagues feed the original input into every iteration (“recall”). The loop then carries a working state alongside a persistent copy of the problem (8). In their experiments, recurrent networks trained on small mazes and similar puzzles solved much larger instances simply by iterating longer. Recall, together with a progressive training routine, kept the solution from degrading as iterations continued. Toggle re-injection in the figure to see where those extra inputs enter.
Once depth is something you can add, it is natural to ask why every token should get the same amount. Adaptive Computation Time (9) and PonderNet (10) learn when to stop iterating. Mixture-of-Depths lets tokens skip blocks (11), and Mixture-of-Recursions learns how many recursions each token gets (12).
Aside: when a richer formulation collapses into a familiar one
A tempting way to “upgrade” a block is to make the messages between positions richer. For example, instead of sending a vector, each position could send a small program: a matrix that transforms the receiver. Before building such a thing, check whether the algebra collapses.
The classic example is fast weights. Let each position write an outer product into a weight matrix, , and let a later position apply to its query :
That is exactly unnormalised linear attention: values weighted by query–key similarity (13). Likewise, many “message modulates receiver” designs reduce to multiplicative gating, which transformers already use in GLU and SwiGLU MLPs (14,15) and in FiLM-style conditioning, which scales and shifts features by a learned function of another input (16). A new notation can leave both the set of representable functions and the ease of learning them unchanged.
The evidence for looping is real but specific. It helps most clearly where a task genuinely needs many sequential steps and where you can iterate longer at test time (7,8). Whether it beats a plain model at a fixed compute budget for ordinary language modelling depends on the budget and on how well the baseline is tuned.
More steps, though, only help if the model can put the result of one step to use in the next. That turns out to be a separate problem.
Knowing is not composing
Suppose a model has learned two facts:
- the author of The Glass Orchard is Mira Kell;
- Mira Kell was born in Port Alder.
Now ask: where was the author of The Glass Orchard born? This is a two-hop question. The first hop finds an intermediate entity, Mira Kell, often called the bridge; the second hop looks something up about it. Remarkably, a model can answer each fact when asked directly and still fail the combination. Press and colleagues measured how often language models answer every sub-question correctly yet still fail the composed question, and called this the compositionality gap. Across the GPT-3 family, single-hop accuracy improved faster with scale than two-hop accuracy, so the gap did not shrink as models grew (17). Lake and Baroni had found related failures of systematic composition in sequence-to-sequence networks (18).
How can a model “know” both facts and still fail? It helps to ask five increasingly demanding questions:
- Does it know the facts? Ask each one on its own.
- Does it compute the bridge? Does a representation of Mira Kell appear inside the network at all?
- Is the bridge readable? Can a simple probe, a small classifier trained to read it out, recover it from a hidden state?
- Is the bridge used? Does the second lookup actually consume it?
- Can the model decide to decompose? Does it know, unprompted, to break the question into steps?
Passing 1 to 3 does not guarantee 4. Biran and colleagues traced how large models process such queries. The bridge is resolved in the earlier layers, and only then is the second hop handled, in later layers. Those later layers may no longer have the knowledge needed to finish the job (19). Their evidence included an intervention called back-patching: copy a hidden state from a later layer into an earlier one, so that the bridge is available sooner. For up to 66% of the queries the model had got wrong, some back-patch produced the right answer. That is a diagnostic intervention, showing what the model can do when helped. It is not a full account of why training left it this way, and it is not the only reason compositions fail. A composed question may also simply be rarer in training data, or the model may never have learned the circuit that connects the two facts.
Here is a deliberately simple model of the “too late” explanation. Pretend the bridge becomes available after layers, and the second lookup needs more layers that can read it. In this cartoon, a single forward pass succeeds only if
that is, both hops must fit, one after the other, inside the layers. Real transformers are messier than this. Computation is spread across layers, overlaps and runs in parallel, and nothing reserves a neat block of layers for each hop. Treat the inequality as an explanatory toy, not a law. Still, it makes one prediction worth having. If you could hand the second hop the bridge as an input, at layer 0, each hop would only need to fit on its own:
With the default settings, each fact fits comfortably, and yet the chained question fails. The model is not missing knowledge; in this cartoon it is missing room to chain. Grow the model until one pass succeeds, or split the question and pay for a second pass. In this cartoon a second pass doubles the cost; in practice it depends on prompt lengths, caching and how much text is generated.
Splitting is easy to write down as code. Run the model twice and pass the first answer into the second prompt:
bridge = model("Who wrote The Glass Orchard?") # pass 1
answer = model(f"Where was {bridge} born?") # pass 2
This is like turning one dense expression into two function calls, where the return value of the first becomes an argument of the second. Self-ask (17) and least-to-most prompting (20) organise reasoning this way. Ordinary chain-of-thought (21) is related but mechanically different. The model generates its intermediate steps as tokens within a single generation, and every later token can attend to those earlier tokens through its own forward pass. Both strategies put intermediate results back into the sequence as inputs, but chain-of-thought does not literally restart the network on a fresh prompt.
Three caveats keep this honest. Someone has to choose the split: a prompt, or training examples that show intermediate steps. Getting a model to discover when and where to split on its own is a much harder problem. Every extra pass or generated token costs compute. And each step can still fail. If every step succeeded independently with probability , a chain of steps would succeed with probability , so 95% per step falls below 60% after ten steps. Real errors are not independent, but compounding is the right worry.
Experiments that train transformers from scratch on synthetic facts tell a consistent story. Wang and colleagues found that models could learn to compose two facts implicitly, but only after very long training. The ability did not extend to systematic out-of-distribution combinations, and their analysis connected this to how the computation was laid out across layers (22). Single-pass composition, in other words, depends not just on what a model knows but on where and when its intermediate results become usable.
If chaining is the hard part, a natural thought is to build the chain out of explicit pieces: reusable modules that a router calls in sequence, like functions.
Can a network learn reusable functions?
GPT-2 is dense: every token goes through the same blocks, and all of each block’s weights are applied to it. Whatever reuse happens is implicit. Modular architectures make it explicit. They keep a pool of modules, usually called experts, and a router that decides which to apply. Mixture-of-experts layers do this per token to scale model capacity cheaply (23,24). Neural module networks assemble a different network for each question from a library of modules (25), and routing networks learn which function blocks to chain (26).
It is worth separating four properties that are easy to conflate, because each claims more than the last:
- Sparse execution: only some parameters run for each input. This cuts arithmetic compared with running every expert, though not necessarily latency, which also depends on routing overhead and memory traffic.
- Specialisation: a module handles a particular kind of input. A tendency, not yet a function.
- Reusable computation: a module implements the same operation wherever it is called.
- Composition: reusable computations chain correctly in new combinations.
None of these implies the next. Csordás and colleagues found that trained networks often implement the same sub-operation with different weights in different contexts, instead of reusing one implementation (27). Mittal and colleagues tested common modular architectures on data with a known modular structure. They found that end-to-end training leaves them well short of their potential, and that optimising them is itself a challenge (28). Why would learning resist modularity? Part of the answer is visible in a two-line calculation.
Take the classic mixture of experts (29,30), where each expert outputs a full probability distribution over answers and a router blends the distributions. Two experts have equal router weights of ½. On some training example, expert 1 gives the correct answer probability 0.05; expert 2 gives it 0.90. The mixture gives
How much does expert 1 learn from this example? Training minimises . Differentiating through the mixture, with the router weights treated as fixed, expert 1 receives exactly the gradient it would get if it were trained alone on this example, multiplied by
The expert that most needs to improve gets 5.3% of the learning signal it would get on its own. The figure opens on this example; drag the sliders to see how quickly the signal vanishes as the expert grows more confidently wrong, or as its partner gets better.
The factor is called expert 1’s responsibility: the fraction of the correct answer’s probability that it supplied. In general, with router weights (which do not depend on the experts’ logits), expert predictions and logits :
Here is the one-hot vector for the correct answer, and is the familiar softmax cross-entropy gradient. Read it as a credit-assignment rule. Experts are taught in proportion to how much of the answer they already explain. Those that already work get more signal, while those that are wrong can go quiet, even though they are exactly the ones that need to change.
Derivation, and how it relates to modern transformer MoE layers
With , , and router weights that do not depend on (the router’s own gradient is a separate term):
The factor is the posterior probability that expert 1 “produced” the answer, the quantity computed in the E-step of the EM algorithm for mixtures (30). If expert 1 puts a logit margin on a wrong class, falls roughly like , and so does its gradient.
This exact formula applies to mixtures of output distributions. The MoE layers in modern transformers usually work differently. A router picks the top- experts for each token, and their hidden-state outputs are summed with gate weights before the rest of the network continues. There, an expert’s gradient is scaled by its gate weight and by the downstream error, and experts that are not selected receive no gradient for that token at all. The details differ, but the same rich-get-richer pressure exists, which is why practical MoE systems add auxiliary load-balancing losses and router noise to keep all experts in use (23,24).
So sparse routing is excellent at scaling capacity per unit of compute, which is why it appears in large models. Whether routed modules become reusable functions that compose in new ways is a separate question, and the evidence says it does not happen automatically (27,28).
Suppose, though, that you did have two good modules, trained separately. Could you simply plug one into the other?
A common language
Imagine training one network that produces vectors and, separately, another that consumes vectors of the same size. Can you connect them? That is what you would need in order to swap components between models, merge separately trained models, or add a skill without retraining everything. GPT-2 never faces this problem, because it trains every layer jointly. Each layer learns to read exactly what the layer before it writes, so their shared format is agreed implicitly. Nothing guarantees that a second training run, even with identical code, would settle on the same format.
Here is the smallest possible example. Two modules each represent two concepts with 2-dimensional vectors:
| concept | module A | module B |
|---|---|---|
| dog | ||
| cat |
Each representation is perfectly good on its own; each module’s own reader decodes it correctly. But if A sends meaning “dog” to B, B reads “cat”. Both are right in their own coordinates and wrong together, like two services that agree on a field name but disagree on its units.
In general, suppose module A writes a vector and its own reader decodes it with . For any invertible matrix ,
In words: if you transform everything the writer produces by and give the reader the inverse transformation first, the pair behaves exactly as before. The dog/cat swap is one such , the permutation . A rotation is another. The training loss of a single module cannot tell these choices apart, so nothing pushes two separately trained modules toward the same one. Real networks need not differ by one clean matrix; their differences can be nonlinear or partial. But this is the simplest way two individually correct modules can end up incompatible. Physicists call a freedom like this a gauge symmetry; engineers might call it an unspecified serialisation format. Hidden units have the same kind of freedom, since they can be permuted without changing the function a network computes (31).
what A sends where B stores each concept
1 / 8 messages understood
The figure implements the two classic remedies:
- Learn a translator. Lenc and Vedaldi joined layers of different networks with a small trained stitching layer (32). Bansal and colleagues showed that good networks of the same architecture, even trained in very different ways, can often be stitched with little loss. Representations learned with more data, width or training time can even be plugged into weaker models to improve them (33). If the mismatch is a pure rotation or reflection, it can be solved in closed form from a few paired examples with orthogonal Procrustes (34). This is the “align from examples” mode.
- Use relative coordinates. Moschella and colleagues describe each vector by its similarities to a fixed set of anchor inputs that both sides can encode. Similarities survive rotations and reflections, so separately trained models can be stitched without any further training (35). This is the “relative to anchors” mode.
Other options include exchanging symbols from a shared discrete codebook instead of continuous vectors (36,37), or explicitly aligning the hidden units of two networks before combining them (31).
The orthogonal Procrustes solution
Given paired vectors (from A) and (from B, the same concepts), we want the rotation or reflection that minimises subject to . Stack the pairs as the columns of and and take the singular value decomposition . The optimum is :
U, S, Vt = np.linalg.svd(B @ A.T)
M = U @ Vt # best orthogonal map from A's frame to B'sIn two dimensions the figure solves the same problem in closed form. With a single pair, a rotation and a reflection fit equally well, so “align from 1 example” can still fail when B’s frame is mirrored.
One distinction matters here. Similar is not the same as interchangeable. Measures such as CKA compare how two representations arrange the same inputs, while ignoring rotations and rescalings (38). A high score suggests similar organisation, but it does not establish that the representations contain the same information, or that one module will work when plugged into the other. That is a functional question, and stitching tests it directly (33). Even then, “compatible after a small adapter is trained” and “interchangeable out of the box” are very different levels of interoperability. In software terms, the first means you can write a translation layer, and the second means both services implement the same API.
Learning more without forgetting
So far we have looked at a network’s computation in one moment. Real systems keep learning, and two questions from outside deep learning are worth borrowing here.
The first is what repeated practice should turn into. Cognitive architectures have long modelled how a frequently used sequence of steps becomes a single step. Anderson’s ACT* theory described knowledge compilation, in which chains of rules merge into one (39), and the Soar architecture chunked the results of problem-solving into new rules (40). In program synthesis, DreamCoder grows a library of reusable abstractions by asking which fragments most compress the programs it has found so far (41). These are explicit, symbolic systems rather than neural networks, but the idea they share is decades old: once a sequence of steps has been practised enough, collapse it into a single, faster step.
The second question is how to learn something new without damaging what is already there. Train a network on task A, then only on task B, and its performance on A often collapses: catastrophic interference (42). The standard remedies come in two families. One protects parameters: elastic weight consolidation penalises changes to weights that mattered for earlier tasks (43). The other protects behaviour. Learning without Forgetting keeps the network’s old outputs stable on new data (44), and experience replay keeps rehearsing a sample of old examples (45).
The two families are not equivalent, because very different weights can compute the same function.
Same function, different parameters: a two-line example
Take a tiny ReLU network . For any , , so
With the weights move a long way while the function does not change at all. Conversely, a small change to a sensitive weight can change the function a lot. Penalising parameter movement and preserving behaviour are therefore different objectives.
Whichever family you choose, you still have to decide whether a change actually made the model better. That turns out to be harder than it sounds.
Does it actually help?
Suppose you have an appealing architectural idea and a modest budget. A good modern testbed is TinyStories, a synthetic corpus of simple children’s stories on which models with only a few million parameters learn to write fluent text (46). Small models train in hours, so you can compare variants and repeat runs with different random seeds.
The hard part is deciding what a fair comparison is. “The same size” can mean the same number of parameters, the same training compute, the same training tokens, the same inference compute per generated token, or the same wall-clock time on real hardware, where memory traffic matters as much as arithmetic. The looped models from earlier make this concrete. They cannot match a plain model on parameters and compute at the same time.
| plain, 8 layers | looped, 4 blocks × 2 | plain, 4 layers | |
|---|---|---|---|
| parameters | 27.3 M | 14.7 M | 14.7 M |
| forward FLOPs / token | 54.5 M | 54.5 M | 29.4 M |
| training FLOPs | 81.8 P | 81.8 P | 44.0 P |
| KV-cache values / token | 8.19 k | 8.19 k | 4.10 k |
Highlighted cells match the plain 8-layer model within 1%. Vocabulary 4,096, tied embeddings; attention over the context is not counted.
Where the numbers come from, and what they leave out
A GPT-2-style block of width has about attention weights (queries, keys, values, output) and MLP weights (two matrices), roughly in total. Multiplying a vector by a matrix costs about 2 FLOPs per weight, so a forward pass costs about the number of weights applied per token. The backward pass costs about twice the forward pass, which gives the familiar estimate of roughly training FLOPs for parameters and tokens (47). A looped model applies each of its weights times per token, so its compute scales with while its parameters scale with . An embedding lookup costs almost no arithmetic, but the output projection costs about FLOPs per token for vocabulary size .
The calculator ignores the attention-over-context term, about extra forward FLOPs per token for context length in Kaplan et al.’s estimate. It also ignores normalisation, nonlinearities and memory traffic, all of which affect wall-clock time.
Two well-known results show why this matters:
- Careful tuning can erase claimed gains. Melis and colleagues found that carefully tuned standard LSTMs outperformed several newer architectures that had been compared against weaker baselines (48).
- Most transformer modifications do not transfer. Narang and colleagues re-implemented dozens of proposed modifications in one codebase. Most did not meaningfully improve performance, and many of those that did were small changes, or came from the codebase they were developed in (49).
The practical lesson is to tune the baseline as hard as the new idea, and to match it on the axis your claim is about. Compute-optimal scaling studies remind us that the right balance between parameters and data itself shifts with the budget (50).
The baseline is also cleverer than it looks. A plain transformer learns many useful mechanisms on its own. ALiBi adds an explicit penalty proportional to token distance to attention scores (51). Yet trained models also discover their own ways to shape attention, such as dumping excess attention onto the first tokens, which act as “attention sinks” (52). An architectural change that hands the model something it would have learned anyway should not be expected to help much. The same logic applies to auxiliary training losses. A loss that succeeds at its own goal has not thereby improved the primary task, and only the primary metric, measured against a well-tuned baseline with the same budget, can tell you that.
Skills in a closed loop
The ideas we have met so far come back in a very physical form when a robot chains skills together. In robotics and reinforcement learning, a skill is a reusable controller for one sub-task, such as reaching a waypoint. Reinforcement learning formalises such temporally extended behaviours as options: a policy together with conditions for where it may start and when it stops (53). Chaining two skills is function composition again: skill A’s final state becomes skill B’s starting state.
Two things are different from text. First, control runs in a closed loop: at every step the controller observes the actual state and reacts, so errors can be corrected as they happen, while an open-loop controller replays a planned sequence and cannot correct anything. Second, the skills read the state in the same physical coordinates, which removes the arbitrary-convention problem of the previous section, at least when the state is fully observed. It does not by itself make two controllers work together.
What remains is the problem of what each skill was trained on. A common way to obtain a skill is behaviour cloning: record an expert’s actions and train a policy to imitate them, as ALVINN did when it learned to steer a vehicle (54). If skill B’s demonstrations all start neatly at the waypoint and at rest, B has never seen what actually happens at a hand-off: skill A typically arrives slightly off target and still moving. The input distribution at the composition boundary differs from the training distribution. This is covariate shift. It is a different interface problem from a coordinate mismatch: B reads the coordinates perfectly well, it has simply never been taught what to do in states like these. Ross and colleagues showed that its errors compound over an episode, and proposed DAgger, which collects expert labels on the states the learner actually visits (55). DART injects noise while recording demonstrations, so they cover nearby states too (56). ALVINN itself was trained on synthetically shifted views so it could recover from being off course (54).
skill A (hand-written) skill B (cloned)
0 / 40 episodes reach the goal
Closed loop: at every step B looks at the current state and copies what the teacher did in the most similar recorded states.
Switch between narrow and broad demonstrations, then turn off feedback. The learning rule never changes; only the coverage of the states B will actually be handed, and its ability to react, decide whether the chain works. These are toy dynamics, not a real robot. Still, they show that the boundary between two composed computations is where things break, here through unfamiliar inputs and not through an unfamiliar format.
What I learned
A few ideas crystallised for me while building these figures:
- A transformer is a fixed-depth program over shared memory. Looping can add depth without adding parameters, but not without adding compute, so every comparison must say what is held equal.
- Knowing is not composing. Facts can be present and an intermediate result even readable, yet unusable in time. Feeding intermediate results back as inputs, by explicit decomposition or by generating them as text, buys depth at a cost, and someone or something must decide where to split.
- Modules are not functions. Sparse execution, specialisation, reuse and composition are different properties, and mixture training’s credit assignment can starve the very components that need to change.
- Interfaces need conventions. Separately trained components can be individually perfect and mutually unintelligible. Similar representations are not automatically interchangeable ones.
- Evaluation is part of the architecture. Tuned baselines and explicit compute matching have overturned many appealing results.
None of these ideas is new, and the papers cited throughout develop each of them much further. What I found valuable was seeing how naturally they map onto things software engineers already reason about: control flow, function calls, data formats, and honest benchmarks.
About the figures and the notation
The figures are small components that compute everything in your browser:
| figure | what it computes |
|---|---|
| stacked vs looped | a parameter and FLOP calculator |
| two-hop | the depth-budget cartoon |
| mixture gradient | the exact responsibility formula |
| coordinate frames | rotations, closed-form Procrustes and cosine profiles on eight 2-D points |
| compute ledger | the standard approximations above |
| hand-off | a point mass integrated with Euler steps, PD controllers, and a 3-nearest-neighbour behaviour-cloning policy |
None of them involves a trained neural network, and none reproduces a measurement from a paper.
Notation.
| symbol | meaning |
|---|---|
| the residual stream, one -dimensional vector per position | |
| number of layers | |
| distinct blocks and loop iterations | |
| and | functions; is composition, applied right to left |
| layers needed for the first and second hop, in the cartoon | |
| router weight, predicted distribution and responsibility of expert | |
| a module’s writer, its reader, and a change of coordinates |