machine learning24 min read

Exploring Transformer Architectures

An interactive exploration of attention, recurrence, and compositional computation through toy models and published research.

Contents
  1. A transformer is a program that writes to shared memory
  2. Depth is a budget
  3. Knowing is not composing
  4. Can a network learn reusable functions?
  5. A common language
  6. Learning more without forgetting
  7. Does it actually help?
  8. Skills in a closed loop
  9. What I learned

If you have written software for a while, you carry a lot of intuition about how programs are organised. You know when to write a function and when to inline it, what breaks when two services disagree about a data format, and why a loop is not the same as pasting a block of code ten times.

When I started looking closely at transformers, I kept reaching for those intuitions. Some transferred beautifully; others broke in instructive ways. This essay follows one question through a handful of published ideas: what kind of computer is a transformer, and what would it take to make it compute differently?

I assume you know GPT-2 (1), the decoder-only descendant of the original transformer (2): token embeddings, a stack of blocks that each contain attention and a feed-forward network (MLP), residual connections, and a final projection back to the vocabulary. Everything else is introduced along the way.

One note before we start. Every interactive figure here is a toy model or an exact calculation that runs in your browser. None of them is a trained neural network, and none shows a recorded experimental measurement. They are there to build intuition; the papers I cite provide the evidence.

A transformer is a program that writes to shared memory

Take one position in a sequence, say the token born in “Where was the author of The Glass Orchard born?”. GPT-2 keeps a vector of width dd for it, and that vector travels upward through the network. Each block reads it, computes something, and adds its result back:

x = embed(tokens)               # (T, d): one vector per position
for block in blocks:            # L blocks, own weights each
    x = x + block.attn(norm(x)) # communicate across positions
    x = x + block.mlp(norm(x))  # compute within each position
logits = unembed(norm(x))

Elhage and colleagues call this vector the residual stream and suggest thinking of it as a shared channel that every layer reads from and writes to (3). To a software engineer it looks like a shared blackboard. Attention is the only operation that moves information between positions, a bit like message passing between workers. The MLP transforms each position in place, like a pure function applied to each worker’s local state.

embeddingunembed → next tokenattnmlpattnmlpattnmlpattnmlpread (grey) · write by addition (blue) · the stream itself is the only shared state
The residual stream as shared state. Each block reads the current vector, computes, and adds its result back. Attention moves information across positions; the MLP transforms each position on its own.

This picture invites a lens I will use throughout: composition of functions. If one layer computes ff, a later one gg and a later one hh, the network computes something like

y=h(g(f(x))),y = h(g(f(x))),

where ff‘s output becomes gg‘s input, and gg‘s output becomes hh‘s input. Two consequences of this humble equation will follow us through the essay. First, hh can only use what gg produced if hh comes after gg, and a transformer has a fixed number of layers to come after. Second, gg has to read ff‘s output the way ff meant it, and in a trained network nobody ever wrote that format down.

The first consequence is the natural place to start: how many steps does a transformer actually get?

Depth is a budget

Some computations come with long chains of dependencies. In the straightforward way to follow a chain of pointers kk times, or to carry digits across a long addition, step i+1i+1 needs the result of step ii. Cleverer parallel algorithms exist for both (pointer jumping, carry-lookahead adders), but they need more total work or extra structure, and a network has to learn them. GPT-2 gives every token exactly LL blocks per forward pass, each with its own weights. So the number of layers fixes two different things at once: how much the model can store (parameters) and how many dependent steps it can execute (sequential depth).

A looped, or recurrent-depth, transformer pulls these apart. It keeps BB distinct blocks and applies the whole group RR times with the same weights, exactly like a loop body:

for r in range(R):          # R iterations: more depth
    for block in blocks:    # only B distinct weight sets
        x = block(x)        # attn + mlp with their residual adds, as above

This is an old and fruitful idea. The Universal Transformer applied one shared block repeatedly, with an optional learned stopping rule (4). ALBERT shared one set of parameters across all of BERT’s layers to shrink the model (5). Looped transformers turn out to be expressive enough to emulate simple programmable computers (6), and recurrent-depth language models can spend more iterations on harder inputs at test time (7). The appeal is clear: if knowledge and sequential steps are separate resources, perhaps you can buy more of one without paying for the other.

Illustrative — no trained model
inputcopyoutputθ1iter 1θ2θ3θ4θ1iter 2θ2θ3θ4θ1iter 3θ2θ3θ412 block applications · 4 parameter sets
block parameters4 × block ≈ 12.6 M
forward block FLOPs / token12 × block ≈ 75.5 M FLOPs
compute ÷ parameters3×
block ≈ 12·d² weights at d = 512; embeddings, output layer and attention over the context not counted
Stacked versus looped blocks. Each cell is one application of a block; its shade identifies which parameter set it uses. Looping multiplies sequential depth and compute without adding parameters. The arithmetic assumes about 12·d² weights per block (the GPT-2 layout) and counts forward matrix-multiply FLOPs only.

Notice what the counters do as you add iterations. Parameters stay at BB blocks, while sequential depth and compute grow to B⋅RB \cdot R blocks. Weight sharing saves parameter memory, not arithmetic: at the same number of block executions, each iteration still performs a full block’s worth of matrix multiplications, and activations and the attention key/value cache still grow with every executed block. That makes “is looping better?” an incomplete question. Compared with a plain model of equal parameters, the looped one gets roughly RR times more block compute, an unfair advantage. Compared with a plain model of equal compute, it has roughly RR times fewer block weights to hold knowledge in. These ratios apply to the transformer blocks; the embeddings and output layer are the same in both. We will need to come back to this when we talk about evaluation.

Loops raise a subtler problem too. Each iteration rewrites the state, so after a few passes the network may have lost track of the original question, like a loop body that receives only the running accumulator and not the original arguments. Bansal and colleagues feed the original input into every iteration (“recall”). The loop then carries a working state alongside a persistent copy of the problem (8). In their experiments, recurrent networks trained on small mazes and similar puzzles solved much larger instances simply by iterating longer. Recall, together with a progressive training routine, kept the solution from degrading as iterations continued. Toggle re-injection in the figure to see where those extra inputs enter.

Once depth is something you can add, it is natural to ask why every token should get the same amount. Adaptive Computation Time (9) and PonderNet (10) learn when to stop iterating. Mixture-of-Depths lets tokens skip blocks (11), and Mixture-of-Recursions learns how many recursions each token gets (12).

Aside: when a richer formulation collapses into a familiar one

A tempting way to “upgrade” a block is to make the messages between positions richer. For example, instead of sending a vector, each position could send a small program: a matrix that transforms the receiver. Before building such a thing, check whether the algebra collapses.

The classic example is fast weights. Let each position jj write an outer product into a weight matrix, W=∑jvjkj⊤W = \sum_j v_j k_j^\top, and let a later position apply WW to its query qq:

Wq=∑jvj (kj⊤q).W q = \sum_j v_j \,(k_j^\top q).

That is exactly unnormalised linear attention: values vjv_j weighted by query–key similarity (13). Likewise, many “message modulates receiver” designs reduce to multiplicative gating, which transformers already use in GLU and SwiGLU MLPs (14,15) and in FiLM-style conditioning, which scales and shifts features by a learned function of another input (16). A new notation can leave both the set of representable functions and the ease of learning them unchanged.

The evidence for looping is real but specific. It helps most clearly where a task genuinely needs many sequential steps and where you can iterate longer at test time (7,8). Whether it beats a plain model at a fixed compute budget for ordinary language modelling depends on the budget and on how well the baseline is tuned.

More steps, though, only help if the model can put the result of one step to use in the next. That turns out to be a separate problem.

Knowing is not composing

Suppose a model has learned two facts:

  • the author of The Glass Orchard is Mira Kell;
  • Mira Kell was born in Port Alder.

Now ask: where was the author of The Glass Orchard born? This is a two-hop question. The first hop finds an intermediate entity, Mira Kell, often called the bridge; the second hop looks something up about it. Remarkably, a model can answer each fact when asked directly and still fail the combination. Press and colleagues measured how often language models answer every sub-question correctly yet still fail the composed question, and called this the compositionality gap. Across the GPT-3 family, single-hop accuracy improved faster with scale than two-hop accuracy, so the gap did not shrink as models grew (17). Lake and Baroni had found related failures of systematic composition in sequence-to-sequence networks (18).

How can a model “know” both facts and still fail? It helps to ask five increasingly demanding questions:

  1. Does it know the facts? Ask each one on its own.
  2. Does it compute the bridge? Does a representation of Mira Kell appear inside the network at all?
  3. Is the bridge readable? Can a simple probe, a small classifier trained to read it out, recover it from a hidden state?
  4. Is the bridge used? Does the second lookup actually consume it?
  5. Can the model decide to decompose? Does it know, unprompted, to break the question into steps?

Passing 1 to 3 does not guarantee 4. Biran and colleagues traced how large models process such queries. The bridge is resolved in the earlier layers, and only then is the second hop handled, in later layers. Those later layers may no longer have the knowledge needed to finish the job (19). Their evidence included an intervention called back-patching: copy a hidden state from a later layer into an earlier one, so that the bridge is available sooner. For up to 66% of the queries the model had got wrong, some back-patch produced the right answer. That is a diagnostic intervention, showing what the model can do when helped. It is not a full account of why training left it this way, and it is not the only reason compositions fail. A composed question may also simply be rarer in training data, or the model may never have learned the circuit that connects the two facts.

Here is a deliberately simple model of the “too late” explanation. Pretend the bridge becomes available after ℓb\ell_b layers, and the second lookup needs ℓr\ell_r more layers that can read it. In this cartoon, a single forward pass succeeds only if

L  ≥  ℓb+ℓr,L \;\ge\; \ell_b + \ell_r ,

that is, both hops must fit, one after the other, inside the LL layers. Real transformers are messier than this. Computation is spread across layers, overlaps and runs in parallel, and nothing reserves a neat block of layers for each hop. Treat the inequality as an explanatory toy, not a law. Still, it makes one prediction worth having. If you could hand the second hop the bridge as an input, at layer 0, each hop would only need to fit on its own:

L  ≥  max⁡(ℓb,ℓr).L \;\ge\; \max(\ell_b, \ell_r).
Illustrative model — not a trained network
fact 1 author(The Glass Orchard) = Mira Kell
fact 2 birthplace(Mira Kell) = Port Alder
question Where was the author of The Glass Orchard born?
one forward passfind "Mira Kell"look up her birthplace✗ second hop runs out of layers
each fact asked aloneboth answered ✓
needs8 + 6 > 12
compute1 pass
A cartoon of the depth-budget argument. Each fact fits inside the model on its own, but chaining them in one pass can run out of layers. Splitting the question into two passes gives each hop the full depth, at the cost of a second pass. An explanatory toy: real networks do not allocate layers this cleanly.

With the default settings, each fact fits comfortably, and yet the chained question fails. The model is not missing knowledge; in this cartoon it is missing room to chain. Grow the model until one pass succeeds, or split the question and pay for a second pass. In this cartoon a second pass doubles the cost; in practice it depends on prompt lengths, caching and how much text is generated.

Splitting is easy to write down as code. Run the model twice and pass the first answer into the second prompt:

bridge = model("Who wrote The Glass Orchard?")  # pass 1
answer = model(f"Where was {bridge} born?")     # pass 2

This is like turning one dense expression into two function calls, where the return value of the first becomes an argument of the second. Self-ask (17) and least-to-most prompting (20) organise reasoning this way. Ordinary chain-of-thought (21) is related but mechanically different. The model generates its intermediate steps as tokens within a single generation, and every later token can attend to those earlier tokens through its own forward pass. Both strategies put intermediate results back into the sequence as inputs, but chain-of-thought does not literally restart the network on a fresh prompt.

Three caveats keep this honest. Someone has to choose the split: a prompt, or training examples that show intermediate steps. Getting a model to discover when and where to split on its own is a much harder problem. Every extra pass or generated token costs compute. And each step can still fail. If every step succeeded independently with probability pp, a chain of kk steps would succeed with probability pkp^k, so 95% per step falls below 60% after ten steps. Real errors are not independent, but compounding is the right worry.

Experiments that train transformers from scratch on synthetic facts tell a consistent story. Wang and colleagues found that models could learn to compose two facts implicitly, but only after very long training. The ability did not extend to systematic out-of-distribution combinations, and their analysis connected this to how the computation was laid out across layers (22). Single-pass composition, in other words, depends not just on what a model knows but on where and when its intermediate results become usable.

If chaining is the hard part, a natural thought is to build the chain out of explicit pieces: reusable modules that a router calls in sequence, like functions.

Can a network learn reusable functions?

GPT-2 is dense: every token goes through the same blocks, and all of each block’s weights are applied to it. Whatever reuse happens is implicit. Modular architectures make it explicit. They keep a pool of modules, usually called experts, and a router that decides which to apply. Mixture-of-experts layers do this per token to scale model capacity cheaply (23,24). Neural module networks assemble a different network for each question from a library of modules (25), and routing networks learn which function blocks to chain (26).

It is worth separating four properties that are easy to conflate, because each claims more than the last:

  1. Sparse execution: only some parameters run for each input. This cuts arithmetic compared with running every expert, though not necessarily latency, which also depends on routing overhead and memory traffic.
  2. Specialisation: a module handles a particular kind of input. A tendency, not yet a function.
  3. Reusable computation: a module implements the same operation wherever it is called.
  4. Composition: reusable computations chain correctly in new combinations.

None of these implies the next. Csordás and colleagues found that trained networks often implement the same sub-operation with different weights in different contexts, instead of reusing one implementation (27). Mittal and colleagues tested common modular architectures on data with a known modular structure. They found that end-to-end training leaves them well short of their potential, and that optimising them is itself a challenge (28). Why would learning resist modularity? Part of the answer is visible in a two-line calculation.

Take the classic mixture of experts (29,30), where each expert outputs a full probability distribution over answers and a router blends the distributions. Two experts have equal router weights of ½. On some training example, expert 1 gives the correct answer probability 0.05; expert 2 gives it 0.90. The mixture gives

p(y)=12(0.05)+12(0.90)=0.475.p(y) = \tfrac12 (0.05) + \tfrac12 (0.90) = 0.475 .

How much does expert 1 learn from this example? Training minimises −log⁡p(y)-\log p(y). Differentiating through the mixture, with the router weights treated as fixed, expert 1 receives exactly the gradient it would get if it were trained alone on this example, multiplied by

r1=12⋅0.050.475≈0.053.r_1 = \frac{\tfrac12 \cdot 0.05}{0.475} \approx 0.053 .

The expert that most needs to improve gets 5.3% of the learning signal it would get on its own. The figure opens on this example; drag the sliders to see how quickly the signal vanishes as the expert grows more confidently wrong, or as its partner gets better.

How much does the struggling expert learn? Exact arithmetic — no training
1e01e-41e-81e-120102030logit margin of the wrong answertrained aloneinside the mixture
expert 1's p(correct)0.050
partner's p(correct)0.900
responsibility r₁ = share of its solo gradient5.3%
gradient size: alone → in mixture1.107 → 0.058
The gradient a struggling expert receives in a mixture of two output distributions: trained alone (dashed) and inside the mixture (solid, log scale). The default reproduces the worked example in the text. As the expert grows more confident in a wrong answer, its share of the credit, and therefore its gradient, collapses, faster still when its partner explains the data well. Exact arithmetic over ten classes; no training involved.

The factor r1r_1 is called expert 1’s responsibility: the fraction of the correct answer’s probability that it supplied. In general, with router weights πk\pi_k (which do not depend on the experts’ logits), expert predictions pkp_k and logits zkz_k:

∂(−log⁡p(y))∂z1  =  π1 p1(y)p(y)⏟r1: responsibility  ⋅  (p1−ey)⏟gradient if trained alone.\frac{\partial(-\log p(y))}{\partial z_1} \;=\; \underbrace{\frac{\pi_1\, p_1(y)}{p(y)}}_{r_1:\ \text{responsibility}} \;\cdot\; \underbrace{\big(p_1 - \mathbf{e}_y\big)}_{\text{gradient if trained alone}} .

Here ey\mathbf{e}_y is the one-hot vector for the correct answer, and p1−eyp_1 - \mathbf{e}_y is the familiar softmax cross-entropy gradient. Read it as a credit-assignment rule. Experts are taught in proportion to how much of the answer they already explain. Those that already work get more signal, while those that are wrong can go quiet, even though they are exactly the ones that need to change.

Derivation, and how it relates to modern transformer MoE layers

With p(y)=∑kπkpk(y)p(y) = \sum_k \pi_k p_k(y), pk=softmax(zk)p_k = \mathrm{softmax}(z_k), and router weights πk\pi_k that do not depend on z1z_1 (the router’s own gradient is a separate term):

∂(−log⁡p(y))∂z1=−π1p(y)∂p1(y)∂z1=−π1p(y) p1(y) (ey−p1)=r1 (p1−ey).\frac{\partial (-\log p(y))}{\partial z_1} = -\frac{\pi_1}{p(y)} \frac{\partial p_1(y)}{\partial z_1} = -\frac{\pi_1}{p(y)}\, p_1(y)\,(\mathbf{e}_y - p_1) = r_1\,(p_1 - \mathbf{e}_y).

The factor r1r_1 is the posterior probability that expert 1 “produced” the answer, the quantity computed in the E-step of the EM algorithm for mixtures (30). If expert 1 puts a logit margin mm on a wrong class, p1(y)p_1(y) falls roughly like e−me^{-m}, and so does its gradient.

This exact formula applies to mixtures of output distributions. The MoE layers in modern transformers usually work differently. A router picks the top-kk experts for each token, and their hidden-state outputs are summed with gate weights before the rest of the network continues. There, an expert’s gradient is scaled by its gate weight and by the downstream error, and experts that are not selected receive no gradient for that token at all. The details differ, but the same rich-get-richer pressure exists, which is why practical MoE systems add auxiliary load-balancing losses and router noise to keep all experts in use (23,24).

So sparse routing is excellent at scaling capacity per unit of compute, which is why it appears in large models. Whether routed modules become reusable functions that compose in new ways is a separate question, and the evidence says it does not happen automatically (27,28).

Suppose, though, that you did have two good modules, trained separately. Could you simply plug one into the other?

A common language

Imagine training one network that produces vectors and, separately, another that consumes vectors of the same size. Can you connect them? That is what you would need in order to swap components between models, merge separately trained models, or add a skill without retraining everything. GPT-2 never faces this problem, because it trains every layer jointly. Each layer learns to read exactly what the layer before it writes, so their shared format is agreed implicitly. Nothing guarantees that a second training run, even with identical code, would settle on the same format.

Here is the smallest possible example. Two modules each represent two concepts with 2-dimensional vectors:

conceptmodule Amodule B
dog[1,0][1, 0][0,1][0, 1]
cat[0,1][0, 1][1,0][1, 0]

Each representation is perfectly good on its own; each module’s own reader decodes it correctly. But if A sends [1,0][1, 0] meaning “dog” to B, B reads “cat”. Both are right in their own coordinates and wrong together, like two services that agree on a field name but disagree on its units.

In general, suppose module A writes a vector ϕ(x)\phi(x) and its own reader decodes it with ψ\psi. For any invertible matrix MM,

ψ(ϕ(x))  =  (ψ M−1)(M ϕ(x)).\psi\big(\phi(x)\big) \;=\; \big(\psi\, M^{-1}\big)\big(M\, \phi(x)\big).

In words: if you transform everything the writer produces by MM and give the reader the inverse transformation M−1M^{-1} first, the pair behaves exactly as before. The dog/cat swap is one such MM, the permutation [0110]\left[\begin{smallmatrix}0&1\\1&0\end{smallmatrix}\right]. A rotation is another. The training loss of a single module cannot tell these choices apart, so nothing pushes two separately trained modules toward the same one. Real networks need not differ by one clean matrix; their differences can be nonlinear or partial. But this is the simplest way two individually correct modules can end up incompatible. Physicists call a freedom like this a gauge symmetry; engineers might call it an unspecified serialisation format. Hidden units have the same kind of freedom, since they can be permuted without changing the function a network computes (31).

Illustrative — toy 2-D vectors
catdogowlfoxelkbeeantyak

what A sends where B stores each concept

1 / 8 messages understood

Two modules that know the same eight concepts, in coordinate systems that differ by a rotation (or, if you tick the box, a mirror image, the general form of the dog/cat swap). Connected directly, B misreads A's messages. Fitting a rotation from a few paired examples, or describing every vector relative to shared anchor concepts, restores communication. Toy 2-D vectors computed live; one paired example cannot tell a rotation from a mirror image.

The figure implements the two classic remedies:

  • Learn a translator. Lenc and Vedaldi joined layers of different networks with a small trained stitching layer (32). Bansal and colleagues showed that good networks of the same architecture, even trained in very different ways, can often be stitched with little loss. Representations learned with more data, width or training time can even be plugged into weaker models to improve them (33). If the mismatch is a pure rotation or reflection, it can be solved in closed form from a few paired examples with orthogonal Procrustes (34). This is the “align from examples” mode.
  • Use relative coordinates. Moschella and colleagues describe each vector by its similarities to a fixed set of anchor inputs that both sides can encode. Similarities survive rotations and reflections, so separately trained models can be stitched without any further training (35). This is the “relative to anchors” mode.

Other options include exchanging symbols from a shared discrete codebook instead of continuous vectors (36,37), or explicitly aligning the hidden units of two networks before combining them (31).

The orthogonal Procrustes solution

Given paired vectors aia_i (from A) and bib_i (from B, the same concepts), we want the rotation or reflection MM that minimises ∑i∥Mai−bi∥2\sum_i \lVert M a_i - b_i \rVert^2 subject to M⊤M=IM^\top M = I. Stack the pairs as the columns of AA and BB and take the singular value decomposition BA⊤=UΣV⊤B A^\top = U \Sigma V^\top. The optimum is M⋆=UV⊤M^\star = U V^\top:

U, S, Vt = np.linalg.svd(B @ A.T)
M = U @ Vt   # best orthogonal map from A's frame to B's

In two dimensions the figure solves the same problem in closed form. With a single pair, a rotation and a reflection fit equally well, so “align from 1 example” can still fail when B’s frame is mirrored.

One distinction matters here. Similar is not the same as interchangeable. Measures such as CKA compare how two representations arrange the same inputs, while ignoring rotations and rescalings (38). A high score suggests similar organisation, but it does not establish that the representations contain the same information, or that one module will work when plugged into the other. That is a functional question, and stitching tests it directly (33). Even then, “compatible after a small adapter is trained” and “interchangeable out of the box” are very different levels of interoperability. In software terms, the first means you can write a translation layer, and the second means both services implement the same API.

Learning more without forgetting

So far we have looked at a network’s computation in one moment. Real systems keep learning, and two questions from outside deep learning are worth borrowing here.

The first is what repeated practice should turn into. Cognitive architectures have long modelled how a frequently used sequence of steps becomes a single step. Anderson’s ACT* theory described knowledge compilation, in which chains of rules merge into one (39), and the Soar architecture chunked the results of problem-solving into new rules (40). In program synthesis, DreamCoder grows a library of reusable abstractions by asking which fragments most compress the programs it has found so far (41). These are explicit, symbolic systems rather than neural networks, but the idea they share is decades old: once a sequence of steps has been practised enough, collapse it into a single, faster step.

The second question is how to learn something new without damaging what is already there. Train a network on task A, then only on task B, and its performance on A often collapses: catastrophic interference (42). The standard remedies come in two families. One protects parameters: elastic weight consolidation penalises changes to weights that mattered for earlier tasks (43). The other protects behaviour. Learning without Forgetting keeps the network’s old outputs stable on new data (44), and experience replay keeps rehearsing a sample of old examples (45).

The two families are not equivalent, because very different weights can compute the same function.

Same function, different parameters: a two-line example

Take a tiny ReLU network y=w2 relu(w1x)y = w_2 \,\mathrm{relu}(w_1 x). For any a>0a > 0, relu(a u)=a relu(u)\mathrm{relu}(a\,u) = a\,\mathrm{relu}(u), so

w2 relu(w1x)  =  w2a relu(a w1x).w_2\,\mathrm{relu}(w_1 x) \;=\; \frac{w_2}{a}\,\mathrm{relu}(a\, w_1 x).

With a=10a = 10 the weights move a long way while the function does not change at all. Conversely, a small change to a sensitive weight can change the function a lot. Penalising parameter movement and preserving behaviour are therefore different objectives.

Whichever family you choose, you still have to decide whether a change actually made the model better. That turns out to be harder than it sounds.

Does it actually help?

Suppose you have an appealing architectural idea and a modest budget. A good modern testbed is TinyStories, a synthetic corpus of simple children’s stories on which models with only a few million parameters learn to write fluent text (46). Small models train in hours, so you can compare variants and repeat runs with different random seeds.

The hard part is deciding what a fair comparison is. “The same size” can mean the same number of parameters, the same training compute, the same training tokens, the same inference compute per generated token, or the same wall-clock time on real hardware, where memory traffic matters as much as arithmetic. The looped models from earlier make this concrete. They cannot match a plain model on parameters and compute at the same time.

Which models are “the same size”? Back-of-the-envelope accounting
plain, 8 layerslooped, 4 blocks × 2plain, 4 layers
parameters27.3 M14.7 M14.7 M
forward FLOPs / token54.5 M54.5 M29.4 M
training FLOPs81.8 P81.8 P44.0 P
KV-cache values / token8.19 k8.19 k4.10 k

Highlighted cells match the plain 8-layer model within 1%. Vocabulary 4,096, tied embeddings; attention over the context is not counted.

Three models and four meanings of “the same size”. With the defaults, the looped model matches the 8-layer plain model in compute but the 4-layer model in parameters: it cannot be matched to both at once. Standard approximations: matrix-multiply FLOPs only, tied embeddings, attention over the context not counted.
Where the numbers come from, and what they leave out

A GPT-2-style block of width dd has about 4d24d^2 attention weights (queries, keys, values, output) and 8d28d^2 MLP weights (two d×4dd \times 4d matrices), roughly 12d212d^2 in total. Multiplying a vector by a matrix costs about 2 FLOPs per weight, so a forward pass costs about 2×2\times the number of weights applied per token. The backward pass costs about twice the forward pass, which gives the familiar estimate of roughly 6ND6ND training FLOPs for NN parameters and DD tokens (47). A looped model applies each of its B⋅12d2B \cdot 12d^2 weights RR times per token, so its compute scales with B⋅RB \cdot R while its parameters scale with BB. An embedding lookup costs almost no arithmetic, but the output projection costs about 2Vd2Vd FLOPs per token for vocabulary size VV.

The calculator ignores the attention-over-context term, about 2 L T d2\,L\,T\,d extra forward FLOPs per token for context length TT in Kaplan et al.’s estimate. It also ignores normalisation, nonlinearities and memory traffic, all of which affect wall-clock time.

Two well-known results show why this matters:

  • Careful tuning can erase claimed gains. Melis and colleagues found that carefully tuned standard LSTMs outperformed several newer architectures that had been compared against weaker baselines (48).
  • Most transformer modifications do not transfer. Narang and colleagues re-implemented dozens of proposed modifications in one codebase. Most did not meaningfully improve performance, and many of those that did were small changes, or came from the codebase they were developed in (49).

The practical lesson is to tune the baseline as hard as the new idea, and to match it on the axis your claim is about. Compute-optimal scaling studies remind us that the right balance between parameters and data itself shifts with the budget (50).

The baseline is also cleverer than it looks. A plain transformer learns many useful mechanisms on its own. ALiBi adds an explicit penalty proportional to token distance to attention scores (51). Yet trained models also discover their own ways to shape attention, such as dumping excess attention onto the first tokens, which act as “attention sinks” (52). An architectural change that hands the model something it would have learned anyway should not be expected to help much. The same logic applies to auxiliary training losses. A loss that succeeds at its own goal has not thereby improved the primary task, and only the primary metric, measured against a well-tuned baseline with the same budget, can tell you that.

Skills in a closed loop

The ideas we have met so far come back in a very physical form when a robot chains skills together. In robotics and reinforcement learning, a skill is a reusable controller for one sub-task, such as reaching a waypoint. Reinforcement learning formalises such temporally extended behaviours as options: a policy together with conditions for where it may start and when it stops (53). Chaining two skills is function composition again: skill A’s final state becomes skill B’s starting state.

Two things are different from text. First, control runs in a closed loop: at every step the controller observes the actual state and reacts, so errors can be corrected as they happen, while an open-loop controller replays a planned sequence and cannot correct anything. Second, the skills read the state in the same physical coordinates, which removes the arbitrary-convention problem of the previous section, at least when the state is fully observed. It does not by itself make two controllers work together.

What remains is the problem of what each skill was trained on. A common way to obtain a skill is behaviour cloning: record an expert’s actions and train a policy to imitate them, as ALVINN did when it learned to steer a vehicle (54). If skill B’s demonstrations all start neatly at the waypoint and at rest, B has never seen what actually happens at a hand-off: skill A typically arrives slightly off target and still moving. The input distribution at the composition boundary differs from the training distribution. This is covariate shift. It is a different interface problem from a coordinate mismatch: B reads the coordinates perfectly well, it has simply never been taught what to do in states like these. Ross and colleagues showed that its errors compound over an episode, and proposed DAgger, which collects expert labels on the states the learner actually visits (55). DART injects noise while recording demonstrations, so they cover nearby states too (56). ALVINN itself was trained on synthetically shifted views so it could recover from being off course (54).

Simulated point mass — illustrative
hand-off Wgoalstart

skill A (hand-written) skill B (cloned)

0 / 40 episodes reach the goal

Closed loop: at every step B looks at the current state and copies what the teacher did in the most similar recorded states.

Chaining two skills on a simulated point mass. Skill A is hand-written and hands over as soon as it is near the waypoint, often still moving. Skill B copies a teacher's actions from its most similar recorded states. Cloned from demonstrations that start at rest, B fails at the hand-off even without disturbances; cloned from broader starts, it succeeds most of the time; without feedback, it fails. A toy simulation in your browser, 40 episodes per setting.

Switch between narrow and broad demonstrations, then turn off feedback. The learning rule never changes; only the coverage of the states B will actually be handed, and its ability to react, decide whether the chain works. These are toy dynamics, not a real robot. Still, they show that the boundary between two composed computations is where things break, here through unfamiliar inputs and not through an unfamiliar format.

What I learned

A few ideas crystallised for me while building these figures:

  • A transformer is a fixed-depth program over shared memory. Looping can add depth without adding parameters, but not without adding compute, so every comparison must say what is held equal.
  • Knowing is not composing. Facts can be present and an intermediate result even readable, yet unusable in time. Feeding intermediate results back as inputs, by explicit decomposition or by generating them as text, buys depth at a cost, and someone or something must decide where to split.
  • Modules are not functions. Sparse execution, specialisation, reuse and composition are different properties, and mixture training’s credit assignment can starve the very components that need to change.
  • Interfaces need conventions. Separately trained components can be individually perfect and mutually unintelligible. Similar representations are not automatically interchangeable ones.
  • Evaluation is part of the architecture. Tuned baselines and explicit compute matching have overturned many appealing results.

None of these ideas is new, and the papers cited throughout develop each of them much further. What I found valuable was seeing how naturally they map onto things software engineers already reason about: control flow, function calls, data formats, and honest benchmarks.

About the figures and the notation

The figures are small components that compute everything in your browser:

figurewhat it computes
stacked vs loopeda parameter and FLOP calculator
two-hopthe depth-budget cartoon
mixture gradientthe exact responsibility formula
coordinate framesrotations, closed-form Procrustes and cosine profiles on eight 2-D points
compute ledgerthe standard approximations above
hand-offa point mass integrated with Euler steps, PD controllers, and a 3-nearest-neighbour behaviour-cloning policy

None of them involves a trained neural network, and none reproduces a measurement from a paper.

Notation.

symbolmeaning
xxthe residual stream, one dd-dimensional vector per position
LLnumber of layers
B,RB, Rdistinct blocks and loop iterations
f,g,hf, g, h and F,G,HF, G, Hfunctions; ∘\circ is composition, applied right to left
ℓb,ℓr\ell_b, \ell_rlayers needed for the first and second hop, in the cartoon
πk,pk,rk\pi_k, p_k, r_krouter weight, predicted distribution and responsibility of expert kk
ϕ,ψ,M\phi, \psi, Ma module’s writer, its reader, and a change of coordinates
1.
Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language Models are Unsupervised Multitask Learners [Internet]. OpenAI; 2019. Available from: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
2.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention Is All You Need. In: Advances in Neural Information Processing Systems [Internet]. 2017. Available from: https://arxiv.org/abs/1706.03762
3.
Elhage N, Nanda N, Olsson C, Henighan T, Joseph N, Mann B, et al. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread [Internet]. 2021; Available from: https://transformer-circuits.pub/2021/framework/index.html
4.
Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł. Universal Transformers. In: International Conference on Learning Representations [Internet]. 2019. Available from: https://arxiv.org/abs/1807.03819
5.
Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In: International Conference on Learning Representations [Internet]. 2020. Available from: https://arxiv.org/abs/1909.11942
6.
Giannou A, Rajput S, Sohn J, Lee K, Lee JD, Papailiopoulos D. Looped Transformers as Programmable Computers. In: International Conference on Machine Learning [Internet]. 2023. Available from: https://arxiv.org/abs/2301.13196
7.
Geiping J, McLeish S, Jain N, Kirchenbauer J, Singh S, Bartoldson BR, et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv preprint arXiv:250205171 [Internet]. 2025; Available from: https://arxiv.org/abs/2502.05171
8.
Bansal A, Schwarzschild A, Borgnia E, Emam Z, Huang F, Goldblum M, et al. End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking. In: Advances in Neural Information Processing Systems [Internet]. 2022. Available from: https://arxiv.org/abs/2202.05826
9.
Graves A. Adaptive Computation Time for Recurrent Neural Networks. In: arXiv preprint arXiv:160308983 [Internet]. 2016. Available from: https://arxiv.org/abs/1603.08983
10.
Banino A, Balaguer J, Blundell C. PonderNet: Learning to Ponder. arXiv preprint arXiv:210705407 [Internet]. 2021; Available from: https://arxiv.org/abs/2107.05407
11.
Raposo D, Ritter S, Richards B, Lillicrap T, Humphreys PC, Santoro A. Mixture-of-Depths: Dynamically Allocating Compute in Transformer-Based Language Models. arXiv preprint arXiv:240402258 [Internet]. 2024; Available from: https://arxiv.org/abs/2404.02258
12.
Bae S, Kim Y, Bayat R, Kim S, others. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. arXiv preprint arXiv:250710524 [Internet]. 2025; Available from: https://arxiv.org/abs/2507.10524
13.
Schlag I, Irie K, Schmidhuber J. Linear Transformers Are Secretly Fast Weight Programmers. In: International Conference on Machine Learning [Internet]. 2021. Available from: https://arxiv.org/abs/2102.11174
14.
Dauphin YN, Fan A, Auli M, Grangier D. Language Modeling with Gated Convolutional Networks. In: International Conference on Machine Learning [Internet]. 2017. Available from: https://arxiv.org/abs/1612.08083
15.
Shazeer N. GLU Variants Improve Transformer. arXiv preprint arXiv:200205202 [Internet]. 2020; Available from: https://arxiv.org/abs/2002.05202
16.
Perez E, Strub F, de Vries H, Dumoulin V, Courville A. FiLM: Visual Reasoning with a General Conditioning Layer. In: AAAI Conference on Artificial Intelligence [Internet]. 2018. Available from: https://arxiv.org/abs/1709.07871
17.
Press O, Zhang M, Min S, Schmidt L, Smith NA, Lewis M. Measuring and Narrowing the Compositionality Gap in Language Models. In: Findings of the Association for Computational Linguistics: EMNLP 2023 [Internet]. 2023. Available from: https://arxiv.org/abs/2210.03350
18.
Lake BM, Baroni M. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In: International Conference on Machine Learning [Internet]. 2018. Available from: https://arxiv.org/abs/1711.00350
19.
Biran E, Gottesman D, Yang S, Geva M, Globerson A. Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing [Internet]. 2024. Available from: https://arxiv.org/abs/2406.12775
20.
Zhou D, Schärli N, Hou L, Wei J, Scales N, Wang X, et al. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In: International Conference on Learning Representations [Internet]. 2023. Available from: https://arxiv.org/abs/2205.10625
21.
Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems [Internet]. 2022. Available from: https://arxiv.org/abs/2201.11903
22.
Wang B, Yue X, Su Y, Sun H. Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization. In: Advances in Neural Information Processing Systems [Internet]. 2024. Available from: https://arxiv.org/abs/2405.15071
23.
Shazeer N, Mirhoseini A, Maziarz K, Davis A, Le Q, Hinton G, et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In: International Conference on Learning Representations [Internet]. 2017. Available from: https://arxiv.org/abs/1701.06538
24.
Fedus W, Zoph B, Shazeer N. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research [Internet]. 2022;23(120):1–39. Available from: https://arxiv.org/abs/2101.03961
25.
Andreas J, Rohrbach M, Darrell T, Klein D. Neural Module Networks. In: IEEE Conference on Computer Vision and Pattern Recognition [Internet]. 2016. Available from: https://arxiv.org/abs/1511.02799
26.
Rosenbaum C, Klinger T, Riemer M. Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning. In: International Conference on Learning Representations [Internet]. 2018. Available from: https://arxiv.org/abs/1711.01239
27.
Csordás R, van Steenkiste S, Schmidhuber J. Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks. In: International Conference on Learning Representations [Internet]. 2021. Available from: https://arxiv.org/abs/2010.02066
28.
Mittal S, Bengio Y, Lajoie G. Is a Modular Architecture Enough? In: Advances in Neural Information Processing Systems [Internet]. 2022. Available from: https://arxiv.org/abs/2206.02713
29.
Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE. Adaptive Mixtures of Local Experts. Neural Computation. 1991;3(1):79–87.
30.
Jordan MI, Jacobs RA. Hierarchical Mixtures of Experts and the EM Algorithm. Neural Computation. 1994;6(2):181–214.
31.
Ainsworth SK, Hayase J, Srinivasa S. Git Re-Basin: Merging Models modulo Permutation Symmetries. In: International Conference on Learning Representations [Internet]. 2023. Available from: https://arxiv.org/abs/2209.04836
32.
Lenc K, Vedaldi A. Understanding Image Representations by Measuring Their Equivariance and Equivalence. In: IEEE Conference on Computer Vision and Pattern Recognition [Internet]. 2015. Available from: https://arxiv.org/abs/1411.5908
33.
Bansal Y, Nakkiran P, Barak B. Revisiting Model Stitching to Compare Neural Representations. In: Advances in Neural Information Processing Systems [Internet]. 2021. Available from: https://arxiv.org/abs/2106.07682
34.
Schönemann PH. A Generalized Solution of the Orthogonal Procrustes Problem. Psychometrika. 1966;31(1):1–10.
35.
Moschella L, Maiorca V, Fumero M, Norelli A, Locatello F, Rodolà E. Relative Representations Enable Zero-Shot Latent Space Communication. In: International Conference on Learning Representations [Internet]. 2023. Available from: https://arxiv.org/abs/2209.15430
36.
van den Oord A, Vinyals O, Kavukcuoglu K. Neural Discrete Representation Learning. In: Advances in Neural Information Processing Systems [Internet]. 2017. Available from: https://arxiv.org/abs/1711.00937
37.
Liu D, Lamb A, Kawaguchi K, Goyal A, Sun C, Mozer MC, et al. Discrete-Valued Neural Communication. In: Advances in Neural Information Processing Systems [Internet]. 2021. Available from: https://arxiv.org/abs/2107.02367
38.
Kornblith S, Norouzi M, Lee H, Hinton G. Similarity of Neural Network Representations Revisited. In: International Conference on Machine Learning [Internet]. 2019. Available from: https://arxiv.org/abs/1905.00414
39.
Anderson JR. Acquisition of Cognitive Skill. Psychological Review. 1982;89(4):369–406.
40.
Laird JE, Rosenbloom PS, Newell A. Chunking in Soar: The Anatomy of a General Learning Mechanism. Machine Learning. 1986;1(1):11–46.
41.
Ellis K, Wong C, Nye M, Sablé-Meyer M, Morales L, Hewitt L, et al. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. In: ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI). 2021.
42.
McCloskey M, Cohen NJ. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In: Psychology of Learning and Motivation. Academic Press; 1989. p. 109–65.
43.
Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, et al. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences [Internet]. 2017;114(13):3521–6. Available from: https://arxiv.org/abs/1612.00796
44.
Li Z, Hoiem D. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence [Internet]. 2018;40(12):2935–47. Available from: https://arxiv.org/abs/1606.09282
45.
Rolnick D, Ahuja A, Schwarz J, Lillicrap TP, Wayne G. Experience Replay for Continual Learning. In: Advances in Neural Information Processing Systems [Internet]. 2019. Available from: https://arxiv.org/abs/1811.11682
46.
Eldan R, Li Y. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? arXiv preprint arXiv:230507759 [Internet]. 2023; Available from: https://arxiv.org/abs/2305.07759
47.
Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, et al. Scaling Laws for Neural Language Models. arXiv preprint arXiv:200108361 [Internet]. 2020; Available from: https://arxiv.org/abs/2001.08361
48.
Melis G, Dyer C, Blunsom P. On the State of the Art of Evaluation in Neural Language Models. In: International Conference on Learning Representations [Internet]. 2018. Available from: https://arxiv.org/abs/1707.05589
49.
Narang S, Chung HW, Tay Y, Fedus W, Févry T, Matena M, et al. Do Transformer Modifications Transfer Across Implementations and Applications? In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing [Internet]. 2021. Available from: https://arxiv.org/abs/2102.11972
50.
Hoffmann J, Borgeaud S, Mensch A, Buchatskaya E, others. Training Compute-Optimal Large Language Models. In: Advances in Neural Information Processing Systems [Internet]. 2022. Available from: https://arxiv.org/abs/2203.15556
51.
Press O, Smith NA, Lewis M. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In: International Conference on Learning Representations [Internet]. 2022. Available from: https://arxiv.org/abs/2108.12409
52.
Xiao G, Tian Y, Chen B, Han S, Lewis M. Efficient Streaming Language Models with Attention Sinks. In: International Conference on Learning Representations [Internet]. 2024. Available from: https://arxiv.org/abs/2309.17453
53.
Sutton RS, Precup D, Singh S. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence. 1999;112(1–2):181–211.
54.
Pomerleau DA. Efficient Training of Artificial Neural Networks for Autonomous Navigation. Neural Computation. 1991;3(1):88–97.
55.
Ross S, Gordon GJ, Bagnell JA. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In: International Conference on Artificial Intelligence and Statistics (AISTATS) [Internet]. 2011. Available from: https://arxiv.org/abs/1011.0686
56.
Laskey M, Lee J, Fox R, Dragan A, Goldberg K. DART: Noise Injection for Robust Imitation Learning. In: Conference on Robot Learning [Internet]. 2017. Available from: https://arxiv.org/abs/1703.09327

← All writing