Removing the reward model

Part 4 produced a closed form for the optimal policy and then discarded it as uncomputable. DPO’s contribution is noticing that it does not need to be computed — only rearranged.

The inversion

Start from eq. 2 of part 4 and solve for the reward instead of the policy.

$$r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$$

The intractable $Z(x)$ is still there. But now substitute this into the Bradley–Terry likelihood from part 3, which depends only on the difference of two rewards for the same prompt — and $\beta \log Z(x)$ is the same in both terms.

It cancels.

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right)\right]$$

That is a supervised loss on preference pairs. Two forward passes through the policy, two through the frozen reference, one logistic loss. No reward model, no value model, no sampling during training, no PPO.

The $\beta$ is the same $\beta$ as the KL coefficient, which is the tell that this is the same objective. DPO has a KL constraint; it just never computes a KL.

What it costs in practice

Four models became one, or one and a bit. And with LoRA the reference is free — disable the adapters and the base weights are $\pi_{\text{ref}}$ — so a 70B DPO run fits where a 7B PPO run struggled. This is most of why DPO displaced PPO across the open ecosystem within about a year.

What it gives up

One thing, and it took the field a while to name it: the data are fixed.

A reward model can score whatever the policy produces right now. A preference dataset was written down before training started, by a model that is not this one, and it cannot say anything about the responses the current policy has drifted towards.

$$\text{PPO: } y \sim \pi_\theta \text{ (now)} \qquad\qquad \text{DPO: } (y_w, y_l) \sim \mathcal{D} \text{ (fixed)}$$

Tajwar et al. show this matters and characterise when. Preference tuning benefits substantially from on-policy data — responses sampled from the current policy — and the gap widens the further the policy moves from whatever generated the dataset. Xu et al. reach a compatible conclusion from the other direction: with sufficient budget and a good reward model, well-tuned PPO still beats DPO on hard tasks, and their explanation is precisely that DPO cannot correct behaviours absent from its data.

The response is iterative DPO: run it, sample from the improved policy, get those new samples labelled, run it again. Which recovers on-policy data at the cost of reintroducing the labelling loop DPO was meant to avoid — and at that point the distance to PPO is smaller than it looked.

Where the argument went next

The interesting turn is that both branches of this series assume the reward is a proxy. Part 3’s overoptimisation, part 4’s leash, this part’s careful $\beta$ — all of it exists because the reward model can be wrong.

When the reward is verifiable — a maths answer that checks out, a test suite that passes — none of that applies. There is nothing to overoptimise into, so the KL leash can be dropped, and R1-style recipes do exactly that with GRPO. What emerges is long chains of reasoning that a leash to the SFT policy would have prevented.

I find this the most interesting thing in post-training right now, and also the most over-generalised. It works because the reward is ground truth. For anything scored by a learned model or an LLM judge, the proxy is back, and so is everything in parts 3 and 4.

Five parts. A base model is doing document continuation; demonstrations redirect it and cannot express a negative; comparisons carry a negative and need to be turned into a scalar; the scalar is only valid near where it was fitted, which is what the KL term is for; and the constrained objective has a closed form that can be inverted, which removes most of the apparatus and one real capability.

What I take from having written it down: almost every design decision in RLHF follows from the reward is learned and therefore wrong somewhere. Read any new method against that sentence and it usually resolves into either a better proxy, a tighter leash, or a way to avoid needing one.