Recurrence

Attention’s quadratic cost is the price of an unbounded lookup. Recurrent formulations fix the state size instead, buying linear scaling and constant-memory decoding at the cost of having to decide, as they go, what is worth remembering.