Learned Absolute Position Embeddings

If the sinusoidal table is a guess at what position should look like, the learned table is a refusal to guess. Allocate $L \times d$ parameters, index by position, add to the token embedding, and let gradient descent decide.

BERT did this, GPT-2 did this, and ViT still does. It performs no worse than sinusoids on any benchmark where both were tried — Vaswani et al. reported the same, and chose sinusoids anyway on the untested hope that they would extrapolate.

The ceiling is structural

$P$ has $L$ rows. Position $L$ has no row. There is no natural value to fall back on, no continuation of a formula, no reasonable default — the model simply cannot be evaluated there. This is why GPT-2 is a 1024-token model in a way that is not a matter of degree: the failure at 1025 is a missing parameter, not a degraded one.

What the table learns

Probing trained tables finds less structure than the parameter count suggests. The dominant component is smooth and low-frequency — neighbouring positions get neighbouring vectors — with the first few positions sitting well apart from everything else, which is the same sink phenomenon seen from the embedding side. Beyond that the table is largely a monotone curve through a handful of dimensions, which is a costly way to encode a scalar.

That observation is the case for the relative schemes. If what the model needs is mostly offset, and the absolute index is a means to it, then encoding the offset directly is both cheaper and unbounded.

import torch
from torch import Tensor, nn


class LearnedPositions(nn.Module):
    def __init__(self, max_len: int, dim: int):
        super().__init__()
        self.table = nn.Embedding(max_len, dim)
        # Small init: position must not dominate content at step zero.
        nn.init.normal_(self.table.weight, std=0.02)

    def forward(self, x: Tensor) -> Tensor:                   # [B, T, D]
        t = x.size(1)
        if t > self.table.num_embeddings:
            raise ValueError(f"{t} tokens exceeds the {self.table.num_embeddings}-row table")
        pos = torch.arange(t, device=x.device)
        return x + self.table(pos)

The explicit raise matters more than it looks. Indexing past the table in PyTorch is a device-side assert that surfaces as an opaque CUDA error several operations later, usually attributed to whatever kernel happened to run next.

Vision keeps it because the objection does not apply: a ViT knows its patch grid at configuration time, and changing resolution is a deliberate act during which the table can be interpolated on the grid. Text has no such structure — token 900 is not spatially between token 899 and 901 in any sense that survives interpolation — so language models moved to rotary embeddings and never came back.