01 — siblings

attention/position/

normalisation/

ffn/sparse/recurrence/

Architectures / normalisation

Normalisation

Normalisation keeps activations in a range the optimiser can work with. Its placement relative to the residual stream matters more than its exact form — pre-norm trains stably and costs a little expressivity, post-norm the reverse. The entries here concern what can be dropped from the standard recipe without losing the stability it buys.

9 entries.

Entries

Batch Normalisationstale

Normalise each feature across the batch; keep running statistics for inference.

DeepNormpromising

Post-norm with a scaled residual and depth-aware initialisation, to a thousand layers.

Dynamic Tanhpromising

Replace the normalisation layer with a scaled tanh; no statistics at all.

Group Normalisationcommon

Normalise over channel groups within one example; batch size drops out.

LayerNormcommon

Re-centre and re-scale each activation vector over its own features.

Pre-Norm and Post-Normstandard

Where the normalisation sits relative to the residual add, and what each choice costs.

QK Normalisationcommon

Normalise queries and keys before the dot product to bound the attention logits.

RMSNormstandard

Re-scaling without re-centring; LayerNorm minus the mean subtraction.

Project onto a learned-radius sphere: one scalar for the whole layer.