Turning comparisons into a number

Part 2 established that we want comparisons rather than demonstrations. This part is about the awkward step in the middle: reinforcement learning needs a scalar reward, and what we have is a pile of “A is better than B”.

Why not just ask for a score

Because people cannot produce one. Ask an annotator to rate a response out of ten and the numbers drift within a session, differ between people who agree completely about the ordering, and compress towards the middle. The information is there and the scale is not.

Comparisons are stable in exactly the way scores are not. So collect those and fit a scalar that explains them.

Bradley–Terry

The model is from 1952 and it assumes each item has a latent quality, with the probability of preferring one being a logistic function of the difference.

$$P(y_w \succ y_l) = \sigma\bigl(r(y_w) - r(y_l)\bigr)$$

Take the negative log likelihood and that is the training objective for a reward model: the base transformer with the unembedding replaced by a scalar head, trained so that preferred responses score higher than rejected ones.

$$\mathcal{L} = -\mathbb{E}\Bigl[\log \sigma\bigl(r_\theta(x, y_w) - r_\theta(x, y_l)\bigr)\Bigr]$$

Only differences appear, so the absolute level is unidentifiable — add a constant to every reward and the loss is unchanged. I found this genuinely disorienting the first time: the reward model’s outputs look like scores and are not. A reward of 3.2 means nothing on its own, cannot be compared across models, and cannot be compared across training runs of the same model.

Then it starts lying

Here is the part I did not expect, and it is the central fact of this series.

The reward model is fitted to finitely many comparisons. It is right in the region where those responses live and unconstrained everywhere else. Policy optimisation is a search for high reward — which is to say, a search for exactly the places where the model is wrong.

Gao et al. measured the shape by training a very large “gold” reward model, optimising against a smaller proxy, and plotting true reward against distance travelled from the starting policy.

$$R_{\text{gold}}(d) = d\,\bigl(\alpha - \beta \log d\bigr), \qquad d = \sqrt{\mathrm{KL}(\pi \,\|\, \pi_{\text{ref}})}$$

Two consequences. The peak is at a finite distance, so past it more optimisation makes the model genuinely worse — while the reported proxy reward continues to climb. And a larger reward model does not merely score better; it tolerates more optimisation before turning, which is the argument for spending compute on the reward model rather than only on the policy.

The hacks are all the same shape

Three reliably observed, and they rhyme.

Length. Annotators prefer longer responses, the reward model learns it, the policy writes essays. Singhal et al. found length explains a large fraction of reward variance, and that some reported RLHF improvements substantially vanish when responses are length-controlled.

Sycophancy. Agreement scores better than correction, so the policy agrees.

Format. Bullet points, headers, confident phrasing — all rewarded independently of content.

Each is a feature that genuinely correlates with quality in the training distribution and is cheap to produce in isolation. Any proxy has such features. The question is not whether they exist but how quickly a search finds them, and the answer is: immediately.

What this forces

Since the proxy is only valid near where it was fitted, the optimisation has to be constrained to stay near there. Not as a stabiliser or a regulariser — as the thing that keeps the objective meaningful.

That constraint is the subject of part 4, and once you see it as a validity condition rather than a training trick, PPO’s design stops looking arbitrary.

Part 4 writes the constrained objective down and follows what it takes to actually optimise it — which turns out to need four models in memory at once, and which explains why part 5 exists.