Architectures / position
Position
Attention is permutation-invariant: shuffle the tokens and the maths does not notice. Position has to be added back by hand, and the choice of how determines whether the model still works at four times the context it was trained on. The schemes here differ mainly in what they do off the end of that training distribution.
10 entries.
Entries
01.02.1O(n²·d)01.02.2O(n·d)01.02.3O(n·d)01.02.4O(n²·d)01.02.5O(n²·d)01.02.6O(n·d)01.02.7O(n·d)01.02.8O(n·d)01.02.9O(n²)01.02.10O(n·d)
ALiBistale
Linear distance penalty on attention scores in place of position embeddings.
Learned Absolute Position Embeddingsstale
A lookup table indexed by position, trained like any other embedding.
Multimodal RoPEcommon
Splitting the rotary dimensions across time, height and width for interleaved image and video input.
NoPEpromising
No positional encoding at all: the causal mask already carries the information.
Relative Position Embeddingsstale
Learned vectors indexed by query–key offset, injected into the score rather than the input.
RoPE Scalingstandard
Extending a rotary model past its training length by rewriting the frequencies.
Rotary Position Embeddingstandard
Relative position injected as a per-pair rotation in complex subspaces.
Sinusoidal Position Encodingstale
Fixed sinusoids of geometrically spaced frequencies, added to the token embeddings.
T5 Relative Position Biascommon
One learned scalar per head per log-spaced distance bucket, added to the logits.
xPospromising
Rotary embeddings with a per-dimension exponential decay, for stable extrapolation.