xPos

Rotary attention fails at long distance in a specific way. The fast dimension pairs, having wrapped many times, contribute what is effectively noise to the score — and because the pairs are summed, that noise is added to whatever the slow pairs were saying. The signal is there; it is buried.

ALiBi solves this by making distance monotonically costly. xPos solves it inside the rotation, by attenuating each pair according to how fast it turns.

The scale cancels into $\zeta^{m-n}$ for the same reason the rotation cancels into $R_{m-n}$: opposite signs on query and key, so only the offset survives. What was a pure oscillation is now a damped one.

Attention resolution

The paper’s contribution is arguably the metric rather than the method. It defines attention resolution — how sharply a head can distinguish adjacent positions — and shows that both RoPE and ALiBi trade it away at long distance, differently.

With $\zeta = 1$ this is RoPE, and the cosines at high $\theta_i$ oscillate without bound as $|m-n|$ grows. With $\zeta < 1$ those terms vanish and the sum converges to the contribution of the slow pairs alone — which is exactly the relative-position signal one wants at distance.

Blockwise causal attention

xPos ships with a second, more pragmatic piece. At inference the sequence is split into blocks of the training length; within a block, attention is normal; across blocks, each query attends to its own block and the one before it. Nothing ever sees an offset larger than the model was trained on.

This is a sliding window with the window snapped to a grid, and it is doing most of the practical extrapolation work in the paper’s evaluation. The decay term is what makes the block boundary unobtrusive rather than a discontinuity.

import torch
from torch import Tensor


def xpos_scale(dim: int, gamma: float = 0.4) -> Tensor:
    """ζ per dimension pair: near 1 for slow pairs, smaller for fast ones."""
    i = torch.arange(0, dim, 2).float() / dim                 # [d/2] in [0, 1)
    return (i + gamma) / (1.0 + gamma)


def apply_xpos(
    q: Tensor, k: Tensor, pos: Tensor, freqs: Tensor, zeta: Tensor
) -> tuple[Tensor, Tensor]:
    """q, k: [B, H, T, D]. Rotation as in RoPE, plus opposite-signed decay."""
    angle = pos[:, None] * freqs[None, :]                     # [T, d/2]
    cos, sin = angle.cos(), angle.sin()

    # Opposite exponents: only ζ^(m-n) survives the dot product.
    decay = zeta[None, :] ** pos[:, None]                     # [T, d/2]
    q = rotate(q, cos, sin) * decay.repeat_interleave(2, -1)
    k = rotate(k, cos, sin) / decay.repeat_interleave(2, -1)
    return q, k

The division on the key side is the numerical hazard. $\zeta^{-n}$ grows without bound, and in fp16 it overflows a few thousand positions in. Production code subtracts a running offset from the exponent so that the decay is computed relative to the current block rather than the start of the sequence — which is the same reason the blockwise scheme exists.

Language models did not adopt it; RoPE scaling reached longer contexts on existing checkpoints, and xPos requires training from scratch. The idea resurfaced in RetNet, where the decay $\zeta^{m-n}$ is precisely what turns attention into a recurrence with a fixed-size state — the retention mechanism is xPos with the softmax removed, and there the decay is not a regulariser but the reason the recurrent form exists at all.