Architectures / attention
Attention
The operation everything else is arranged around. A token asks every other token a question, weights the answers by how well they match, and sums. The cost of that generosity is quadratic in sequence length, so most of the work here is about giving some of it up on purpose — restricting who may be asked, or sharing the machinery that stores the answers.
11 entries, 2 of them stubs.
Entries
01.01.1O(n·m·d)01.01.2—01.01.3O(n·(w+g)·d)01.01.4O(n²)01.01.5O(n·m·d)01.01.6O(n²·d)01.01.7O(n·d²)01.01.8O(n²·d)01.01.9O(n²·d)01.01.10O(n²·d)01.01.11O(n·w·d)
Additive Attentionstale
The original mechanism: a learned MLP scores each query–key pair.
Attention Sinksstandard
Softmax must sum to one, so heads need somewhere to dump mass; evict it and generation collapses.
Block-Sparse Attentioncommon
Local windows plus a few global tokens, chosen before the scores are computed.
Causal Maskingstandard
The triangular mask that turns one forward pass into n training examples.
Cross-Attentioncommon
Queries from one sequence, keys and values from another.
Grouped-Query Attentionstandard
g key–value heads shared across h query heads; the dial between MHA and MQA.
Linear Attentionpromising
Drop the softmax, reassociate the matmuls, and decode in constant memory.
Multi-Head Attentionstandard
h attention operations in d/h-dimensional subspaces, summed back into the residual stream.
Multi-Head Latent Attentionstubpromising
Cache a shared low-rank latent instead of per-head keys and values.
Scaled Dot-Product Attentionstandard
Content-based retrieval over key–value pairs with 1/√d_k logit scaling.
Sliding-Window Attentionstubcommon
Local band mask; receptive field grows linearly with depth.