01 — siblings

attention/position/normalisation/ffn/sparse/

recurrence/

Architectures / recurrence

Recurrence

Attention’s quadratic cost is the price of an unbounded lookup. Recurrent formulations fix the state size instead, buying linear scaling and constant-memory decoding at the cost of having to decide, as they go, what is worth remembering.

8 entries.

Entries

The initialisation that makes a linear recurrence remember: online polynomial projection.

O(n·d·N)01.06.2

Hybrid Recurrent-Attentioncommon

A few attention layers among many recurrent ones, because the ratio beats either extreme.

O(n·d + n²·d/r)01.06.3

LSTM and GRUstale

Gates and an additive cell path, so the gradient has somewhere to travel.

O(n·d²)01.06.4

Parallel Scanstandard

Compute a linear recurrence in log-depth instead of a loop.

O(n·log n)01.06.5

Recurrent Neural Networksstale

One hidden state, carried forward one step at a time, and why the gradient dies.

O(n·d²)01.06.6

A transformer-shaped block whose token mixer is an exponentially decaying sum.

Selective State Space Modelspromising

Linear recurrence with input-dependent dynamics; Mamba and the S4 line.

O(n·d·N)01.06.8

State Space Dualitypromising

Selective SSMs and linear attention are the same operator, written two ways.