AI Grimoire
02 — siblings
objectives/
optimisers/scaling/memory/distributed/

Training / objectives

Objectives

Pre-training has one objective and post-training has many. These entries cover the alignment losses — the policy-gradient methods and the closed-form preference objectives that replaced them — and the arguments about which parts of the reinforcement-learning apparatus were ever load-bearing.

3 entries.

Entries

02.01.1
Direct Preference Optimisationstandard
Closed-form preference objective that removes the reward model.
O(d)
02.01.2
GRPOcommon
Group-relative advantages from sampled completions; no value network.
O(G·T·d)
02.01.3
Proximal Policy Optimisationcommon
Clipped surrogate advantage objective with KL control to a reference.
O(T·d)