Attention Sinks

A softmax row sums to one. A head that has nothing to retrieve at a given position cannot simply attend to nothing — it has to put the mass somewhere. What it learns to do is put it on a position whose value vector is close to useless, and the earliest tokens are the natural candidates: every query in the sequence can see them, so they are the only positions available to all rows.

The effect is large and it is not subtle. In a trained decoder, the first token routinely receives more than half the attention mass in most heads of most layers, and it does so whether that token is a <bos> marker, a full stop, or a word chosen at random.

Why sliding windows break

A sliding window evicts the oldest tokens as generation proceeds. Once the window has moved past position 0, the sink is gone, and the mass that used to land there is redistributed over the tokens that remain — tokens the head had no intention of reading. Perplexity does not degrade gracefully; it explodes, typically the moment the first token leaves the cache.

The fix is almost embarrassingly cheap: keep the first four tokens pinned in the cache permanently and slide the window over everything after them. No retraining, no architectural change, and streaming generation over millions of tokens becomes stable.

The same thing under other names

Read as “the softmax cannot express no-op”, the phenomenon connects to two others. Quantisation researchers found the same heads produce extreme activation outliers, precisely because forcing mass onto a useless position requires a large value to be scaled down — which is what makes those layers hard to quantise. And the proposed remedy is the same in both literatures: give the softmax a denominator term that belongs to no position at all.

With that extra $1$, a row may sum to less than one, and a head that wants to do nothing can. Models trained this way show neither sinks nor the associated outliers — but it is a pre-training change, which is why the pinned-token workaround is what gets used on models that already exist.

import torch
from torch import Tensor


class SinkCache:
    """Sliding window that never evicts the first `n_sink` positions."""

    def __init__(self, n_sink: int = 4, window: int = 4096) -> None:
        self.n_sink, self.window = n_sink, window
        self.k: Tensor | None = None
        self.v: Tensor | None = None

    def append(self, k: Tensor, v: Tensor) -> tuple[Tensor, Tensor]:  # [B, H, T, D]
        self.k = k if self.k is None else torch.cat([self.k, k], dim=-2)
        self.v = v if self.v is None else torch.cat([self.v, v], dim=-2)

        if self.k.size(-2) > self.n_sink + self.window:
            keep = self.k.size(-2) - self.window
            # Sinks, then the tail. The gap in between is simply dropped.
            self.k = torch.cat([self.k[..., : self.n_sink, :], self.k[..., keep:, :]], dim=-2)
            self.v = torch.cat([self.v[..., : self.n_sink, :], self.v[..., keep:, :]], dim=-2)

        return self.k, self.v

One detail decides whether this works: positions must be assigned by index within the cache, not by absolute position in the stream. With RoPE that means applying the rotation at attention time rather than when the key is first written, so the sinks stay adjacent to the window instead of drifting arbitrarily far from it. Cache the unrotated keys.