Rotary Position Embedding

Position enters through a rotation applied to queries and keys, chosen so that the dot product between two positions depends only on their offset.

[position][attention]Current standard

Standing

Current standardPart of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.

Effectively unanimous. The live questions are about extending it — base frequency, interpolation, per-dimension scaling — not about replacing it.

judged as of 2026-09 · what the labels mean

Theory

Split each $d$ -dimensional query into $d/2$ planes. RoPE rotates plane $i$ at position $m$ by angle $m\theta_i$ , with $\theta_i = b^{-2i/d}$ and base $b$ typically $10^4$ . Because rotations compose, the inner product between a rotated query at $m$ and a rotated key at $n$ collapses to a function of $m - n$ .

\langle R_m q,\; R_n k \rangle = \langle q,\; R_{n-m} k \rangle

The low-frequency planes rotate slowly and carry long-range order; the high-frequency ones alias almost immediately. Context extension methods exploit exactly this split — position interpolation divides every angle by the extension factor, while NTK-aware scaling raises the base instead so that high frequencies are left alone.

\theta_i = b^{-2i/d}, \qquad b' = b \cdot s^{\,d/(d-2)}

Implementation

python · torch ≥ 2.1

import torch
from torch import Tensor


def rope_cache(seq_len: int, dim: int, base: float = 10_000.0):
    """Precompute cos/sin tables of shape [seq_len, dim // 2]."""
    inv_freq = base ** -(torch.arange(0, dim, 2).float() / dim)
    t = torch.arange(seq_len).float()
    angles = torch.outer(t, inv_freq)
    return angles.cos(), angles.sin()


def apply_rope(x: Tensor, cos: Tensor, sin: Tensor) -> Tensor:
    """x: [B, H, N, D] with D even. Rotates adjacent pairs."""
    x1, x2 = x[..., 0::2], x[..., 1::2]
    cos, sin = cos[None, None], sin[None, None]
    out = torch.stack(
        (x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1
    )
    return out.flatten(-2)

Build the tables in fp32 and cast at the point of use — computing the angles in fp16 loses enough precision at long context to visibly degrade retrieval.

rope-scaling sinusoidal-position-encoding multimodal-rope xpos scaled-dot-product-attention flash-attention

References

[1]Su et al. — RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)arXiv:2104.09864

[2]Chen et al. — Extending Context Window via Position Interpolation (2023)arXiv:2306.15595

[3]Press et al. — Train Short, Test Long: ALiBi (2021)arXiv:2108.12409