Tree — 01

attention/11 position/10 normalisation/9 ffn/8 sparse/8 recurrence/8

Architectures

The shapes a forward pass can take. Attention and its cheaper cousins, how position is injected when the operation itself is permutation-invariant, where normalisation goes and what breaks when it goes elsewhere, and the conditional-compute blocks that buy parameters without buying FLOPs.

54 entries across six branches. Two are stubs and are marked as such — read them as open questions rather than answers.

Attention

the quadratic core, and the windows and sinks that cheapen it

Additive Attentionstale

The original mechanism: a learned MLP scores each query–key pair.

Attention Sinksstandard

Softmax must sum to one, so heads need somewhere to dump mass; evict it and generation collapses.

Block-Sparse Attentioncommon

Local windows plus a few global tokens, chosen before the scores are computed.

O(n·(w+g)·d)01.4

Causal Maskingstandard

The triangular mask that turns one forward pass into n training examples.

Cross-Attentioncommon

Queries from one sequence, keys and values from another.

Grouped-Query Attentionstandard

g key–value heads shared across h query heads; the dial between MHA and MQA.

Linear Attentionpromising

Drop the softmax, reassociate the matmuls, and decode in constant memory.

Multi-Head Attentionstandard

h attention operations in d/h-dimensional subspaces, summed back into the residual stream.

Multi-Head Latent Attentionstubpromising

Cache a shared low-rank latent instead of per-head keys and values.

Scaled Dot-Product Attentionstandard

Content-based retrieval over key–value pairs with 1/√d_k logit scaling.

Sliding-Window Attentionstubcommon

Local band mask; receptive field grows linearly with depth.

Position

injecting order into an operation that has none

Linear distance penalty on attention scores in place of position embeddings.

Learned Absolute Position Embeddingsstale

A lookup table indexed by position, trained like any other embedding.

Multimodal RoPEcommon

Splitting the rotary dimensions across time, height and width for interleaved image and video input.

No positional encoding at all: the causal mask already carries the information.

Relative Position Embeddingsstale

Learned vectors indexed by query–key offset, injected into the score rather than the input.

RoPE Scalingstandard

Extending a rotary model past its training length by rewriting the frequencies.

Rotary Position Embeddingstandard

Relative position injected as a per-pair rotation in complex subspaces.

Sinusoidal Position Encodingstale

Fixed sinusoids of geometrically spaced frequencies, added to the token embeddings.

T5 Relative Position Biascommon

One learned scalar per head per log-spaced distance bucket, added to the logits.

Rotary embeddings with a per-dimension exponential decay, for stable extrapolation.

Normalisation

normalisation/ ›

where the rescaling goes, and what breaks when it moves

Batch Normalisationstale

Normalise each feature across the batch; keep running statistics for inference.

DeepNormpromising

Post-norm with a scaled residual and depth-aware initialisation, to a thousand layers.

Dynamic Tanhpromising

Replace the normalisation layer with a scaled tanh; no statistics at all.

Group Normalisationcommon

Normalise over channel groups within one example; batch size drops out.

LayerNormcommon

Re-centre and re-scale each activation vector over its own features.

Pre-Norm and Post-Normstandard

Where the normalisation sits relative to the residual add, and what each choice costs.

QK Normalisationcommon

Normalise queries and keys before the dot product to bound the attention logits.

RMSNormstandard

Re-scaling without re-centring; LayerNorm minus the mean subtraction.

Project onto a learned-radius sphere: one scalar for the whole layer.

Feed-forward

the position-wise block that holds most of the parameters

Activation Sparsitypromising

Most FFN neurons are zero for any given token; skip the rows they own.

O(n·d·s)01.32

Feed-Forward as Key-Value Memorycommon

Read the FFN as a lookup table: W₁ rows are patterns, W₂ columns are what gets written.

Feed-Forward Networksstandard

Two matrices and a nonlinearity, applied to each position alone.

A smooth gate on the identity, and the activation lineage it belongs to.

Parallel Attention and FFNcommon

Run both branches off the same input instead of one after the other.

Product-Key Memorypromising

A million-entry lookup layer searched in √N time by factorising the key space.

O(√N·d)01.37

Squared ReLUpromising

ReLU, then square: cheaper than a gate and far sparser than either.

Gated feed-forward block; a Swish gate multiplied into a linear branch.

Sparse blocks

conditional compute: more parameters, the same FLOPs

Expert Capacity and Token Droppingcommon

Fixed buffers per expert, and what happens to the tokens that do not fit.

Expert Parallelismstandard

Experts on separate devices, and the two all-to-alls that connect them.

Expert-Choice Routingpromising

Invert the assignment: each expert picks its top tokens, not the reverse.

Loss-Free Load Balancingcommon

Balance experts with a per-expert bias on the router, not with a gradient.

Mixture-of-Depthspromising

Route tokens past whole blocks: conditional depth rather than conditional width.

O(c·n²·d)01.44

Mixture-of-Experts Routingstandard

Top-k gate over expert FFNs with auxiliary load-balancing loss.

Shared and Fine-Grained Expertsstandard

Many small experts, plus a few that every token uses.

O(k·d²/m)01.46

Sparse Upcyclingcommon

Turn a finished dense checkpoint into a mixture of experts by copying the FFN.

Recurrence

recurrence/ ›

linear-time sequence models and what they trade away

The initialisation that makes a linear recurrence remember: online polynomial projection.

O(n·d·N)01.48

Hybrid Recurrent-Attentioncommon

A few attention layers among many recurrent ones, because the ratio beats either extreme.

O(n·d + n²·d/r)01.49

LSTM and GRUstale

Gates and an additive cell path, so the gradient has somewhere to travel.

Parallel Scanstandard

Compute a linear recurrence in log-depth instead of a loop.

O(n·log n)01.51

Recurrent Neural Networksstale

One hidden state, carried forward one step at a time, and why the gradient dies.

A transformer-shaped block whose token mixer is an exponentially decaying sum.

Selective State Space Modelspromising

Linear recurrence with input-dependent dynamics; Mamba and the S4 line.

O(n·d·N)01.54

State Space Dualitypromising

Selective SSMs and linear attention are the same operator, written two ways.