Training / objectives
Objectives
Pre-training has one objective and post-training has many. These entries cover the alignment losses — the policy-gradient methods and the closed-form preference objectives that replaced them — and the arguments about which parts of the reinforcement-learning apparatus were ever load-bearing.
3 entries.
Entries
02.01.1O(d)02.01.2O(G·T·d)02.01.3O(T·d)
Direct Preference Optimisationstandard
Closed-form preference objective that removes the reward model.
GRPOcommon
Group-relative advantages from sampled completions; no value network.
Proximal Policy Optimisationcommon
Clipped surrogate advantage objective with KL control to a reference.