Tree — 02

objectives/8 optimisers/8 scaling/8 memory/8 distributed/8

Training

Everything between random initialisation and a checkpoint worth keeping. Memory arithmetic that decides what fits, sharding strategies that decide what scales, and the preference objectives that have largely replaced hand-written reward functions.

40 entries across five branches. One is a stub and is marked as such — read them as open questions rather than answers.

Objectives

objectives/ ›

what the loss actually asks the model to do

Direct Preference Optimisationstandard

Closed-form preference objective that removes the reward model.

Group-relative advantages from sampled completions; no value network.

KL Regularisationstandard

A leash to the reference policy, and the three ways it is estimated.

Knowledge Distillationcommon

Train the small model on the big one’s distribution, not on the labels.

Masked Language Modellingcommon

Corrupt the input, predict what was removed; the objective that lost to next-token.

Next-Token Predictionstandard

Cross-entropy over the next token, and everything that follows from teacher forcing.

Proximal Policy Optimisationcommon

Clipped surrogate advantage objective with KL control to a reference.

Reward Modellingstandard

Fit a scalar reward to pairwise preferences, then watch the policy exploit it.

Optimisers

optimisers/ ›

the update rule and the schedule wrapped around it

Adaptive moments with weight decay decoupled from the gradient.

Gradient Clippingstandard

Bound the update norm so one bad batch cannot destroy a run.

Learning-Rate Schedulesstandard

Warmup, cosine decay, and the warmup–stable–decay alternative.

Maximal Update Parameterisationcommon

Scale the initialisation and learning rates so the optimum transfers across width.

Memory-Efficient Optimiserscommon

Adam’s state is twice the model; factorise it, quantise it, or project it.

Schedule-Free Optimisationpromising

Averaging in place of a decay schedule, so the run has no fixed horizon.

Second-Order Preconditioningstubpromising

Shampoo, Muon and the orthogonalised-update family.

Stochastic Gradient Descentcommon

A step against the gradient, momentum on top, and why it loses to Adam on transformers.

Scaling

how loss falls as compute, data and parameters grow

Chinchilla Scaling Lawscommon

Compute-optimal allocation between parameters and training tokens.

Compute-Optimal versus Inference-Optimalstandard

Chinchilla minimises training cost. Nobody trains a model they only train once.

Data-Constrained Scalingcommon

What repeated tokens are worth, and how many epochs before they are worth nothing.

Emergent Abilitiescommon

Capabilities that appear abruptly with scale, and the argument that the abruptness is the metric.

Kaplan Scaling Lawsstale

The 2020 power laws, and the three choices that made them recommend the wrong model.

Scaling Law Methodologycommon

How the curves are fitted, and how the fits go wrong.

Test-Time Compute Scalingstandard

Spend more compute per question instead of more compute per model.

Training Compute Accountingstandard

Why a training run costs 6ND FLOPs, and where the six comes from.

Memory

trading arithmetic for activation storage

Activation Compressionpromising

Store the saved activations in fewer bits instead of recomputing them.

O(L·B·T·d)02.26

Activation Memory Accountingstandard

How much a forward pass must keep, and which term dominates when.

O(L·B·T·d)02.27

CPU and NVMe Offloadingcommon

Park what is not in use on host memory or disk, bounded by PCIe rather than compute.

Gradient Accumulationstandard

Simulate a large batch with several small ones, and the normalisation bug that follows.

Gradient Checkpointingstandard

Trade recompute for activation memory at segment boundaries.

Memory Fragmentationcommon

The allocator that says out-of-memory with gigabytes free, and what to do about it.

Mixed-Precision Trainingstandard

Half-precision compute, full-precision accumulation, and the master copy in between.

Selective Recomputationcommon

Recompute only the cheap-and-large activations, not the whole block.

Distributed

distributed/ ›

splitting a model that does not fit on one device

Collective Communicationstandard

All-reduce, all-gather, reduce-scatter, all-to-all — the six primitives everything is built from.

Context Parallelismcommon

Split the sequence across devices, and pass the KV blocks around a ring.

O(n²·d/N)02.35

Data Parallelismstandard

Replicate the model, split the batch, average the gradients.

Fault Tolerancecommon

At ten thousand devices something fails hourly; the run has to survive it.

Parallelism Compositionstandard

Four or five strategies at once, and the topology decides the order.

Pipeline Parallelismstandard

Layers on different devices, and the bubble that schedule design exists to shrink.

Tensor Parallelismstandard

Split each matrix across devices; two all-reduces per layer buy the memory back.

ZeRO Shardingstandard

Partition optimiser state, gradients and parameters across ranks.