Sinusoidal Position Encoding

Self-attention is a sum over a set. Permute the tokens and every score, every weight and every output permutes with them — the operation genuinely cannot tell the cat sat from sat cat the. Position is not something attention loses; it is something attention never had, and it has to be supplied from outside.

The original transformer supplies it by adding a fixed vector to each token embedding before the first layer.

Read it as a clock face with $d/2$ hands. The hand at $i = 0$ turns once per token; the hand at $i = d/2 - 1$ turns once per $2\pi \cdot 10^4$ tokens. Any position is the unique reading of all the hands at once, and nearby positions give nearby readings on the fast hands while agreeing on the slow ones.

The relative-position argument

The paper’s justification is that offsets are linear: for a fixed $k$ there is a matrix $M_k$, independent of $p$, with $\mathrm{PE}_{p+k} = M_k \,\mathrm{PE}_p$. It is a rotation by $k\omega_i$ in each two-dimensional subspace, so the claim is exactly true.

What does not follow is that the model gets relative position for free. The encoding is added to the content vector, so a query at position $p$ is $W_Q(x_p + \mathrm{PE}_p)$ and the score expands into four terms — content with content, content with position, position with content, position with position. Only the last depends on the offset alone, and it is entangled with the other three in a single scalar. The linear structure is available to be learned; it is not imposed.

Relative position embeddings exist because of that expansion, and rotary embeddings exist because rotating $q$ and $k$ rather than adding to $x$ makes the offset dependence structural instead of learned.

Why it does not extrapolate

Nothing about the table is undefined past the training length — the sinusoids carry on. The problem is that the scores do not. Positions beyond anything seen in training produce combinations of phases the attention weights were never fit against, and perplexity degrades sharply within a few hundred tokens of the training window. The encoding is defined everywhere; the model is not.

import math

import torch
from torch import Tensor


def sinusoidal(n: int, d: int, base: float = 10_000.0) -> Tensor:
    """[n, d] table. Even dims sine, odd dims cosine, shared frequency."""
    p = torch.arange(n).unsqueeze(1)                          # [n, 1]
    i = torch.arange(0, d, 2)                                 # [d/2]
    # exp/log rather than a direct power: base**(i/d) overflows fp32 for large d.
    freq = torch.exp(-math.log(base) * i / d)                 # [d/2]

    pe = torch.zeros(n, d)
    pe[:, 0::2] = torch.sin(p * freq)
    pe[:, 1::2] = torch.cos(p * freq)
    return pe

Two conventions are in circulation and they disagree. The paper interleaves sine and cosine on adjacent dimensions, as above; several widely copied implementations concatenate the sine block and the cosine block instead. Both train equally well, and neither is compatible with the other’s checkpoints — the same silent-mismatch hazard that later reappears in RoPE’s two pairing schemes.