02 — siblings

objectives/

optimisers/scaling/memory/distributed/

Training / objectives

Objectives

Pre-training has one objective and post-training has many. These entries cover the alignment losses — the policy-gradient methods and the closed-form preference objectives that replaced them — and the arguments about which parts of the reinforcement-learning apparatus were ever load-bearing.

8 entries.

Entries

Direct Preference Optimisationstandard

Closed-form preference objective that removes the reward model.

Group-relative advantages from sampled completions; no value network.

O(G·T·d)02.01.3

KL Regularisationstandard

A leash to the reference policy, and the three ways it is estimated.

Knowledge Distillationcommon

Train the small model on the big one’s distribution, not on the labels.

Masked Language Modellingcommon

Corrupt the input, predict what was removed; the objective that lost to next-token.

O(n·V·m)02.01.6

Next-Token Predictionstandard

Cross-entropy over the next token, and everything that follows from teacher forcing.

Proximal Policy Optimisationcommon

Clipped surrogate advantage objective with KL control to a reference.

Reward Modellingstandard

Fit a scalar reward to pairwise preferences, then watch the policy exploit it.