Tree — 01
attention/2
normalisation/1
position/1
sparse/1
Architectures
The shapes a forward pass can take. Attention and its cheaper cousins, how position is injected when the operation itself is permutation-invariant, where normalisation goes and what breaks when it goes elsewhere, and the conditional-compute blocks that buy parameters without buying FLOPs.
Five entries. One is a stub and is marked as such — read them as open questions rather than answers.
Entries
01.1O(k·d²)01.2O(n·d)01.3O(n·d)01.4O(n²·d)01.5O(n·w·d)
Mixture-of-Experts Routing
Top-k gate over expert FFNs with auxiliary load-balancing loss.
RMSNorm
Re-scaling without re-centring; LayerNorm minus the mean subtraction.
Rotary Position Embedding
Relative position injected as a per-pair rotation in complex subspaces.
Scaled Dot-Product Attention
Content-based retrieval over key–value pairs with 1/√d_k logit scaling.
Sliding-Window Attentionstub
Local band mask; receptive field grows linearly with depth.