AI Grimoire
01 — siblings
attention/position/
normalisation/
ffn/sparse/recurrence/

Architectures / normalisation

Normalisation

Normalisation keeps activations in a range the optimiser can work with. Its placement relative to the residual stream matters more than its exact form — pre-norm trains stably and costs a little expressivity, post-norm the reverse. The entries here concern what can be dropped from the standard recipe without losing the stability it buys.

1 entry.

Entries

01.03.1
RMSNormstandard
Re-scaling without re-centring; LayerNorm minus the mean subtraction.
O(n·d)