Where demonstrations run out

Part 1 ended with two ways to ask a human what they want. This part takes the first one — show me — as far as it goes.

It is just more pre-training

Supervised fine-tuning has no new machinery. Same objective, same optimiser, different data: prompt-and-response pairs instead of documents, with the loss masked so it only applies to the response.

That mask is the whole implementation and it is easy to get wrong. Include the prompt tokens and you are training the model to generate questions as well as answers, which degrades it in a way that produces a perfectly normal-looking loss curve. I have made this mistake; the symptom was a model that would occasionally continue past its answer into a plausible next user turn.

It works alarmingly well

LIMA is the result that reset my expectations. A 65B base model, one thousand carefully written examples, no reinforcement learning of any kind — and outputs competitive with heavily post-trained models in human comparisons.

A thousand. Not a hundred thousand.

Their reading is the superficial alignment hypothesis: essentially all capability arrives during pre-training, and post-training is teaching the model which of its existing distributions to sample from. If that is right, then a small number of examples suffices, because you are selecting a mode rather than teaching a skill — and the evidence is consistent with it, since quality of examples matters far more than quantity in every ablation I have seen.

The three ways it stops

It cannot express a negative. A demonstration says “produce this”. It never says “not that”. If the model has a failure mode — hedging, over-refusing, inventing citations — you can show it a thousand examples of not doing that, and each one is only positive evidence for one alternative. The gradient never points away from the failure, because the failure never appears in the data.

The second term on the right is the one SFT structurally cannot have.

The ceiling is the demonstrator. The model is fitted to imitate the distribution of responses in the data, so it converges towards the quality of whoever wrote them. That is fine when the writers are excellent and expensive. It is a hard bound when they are not — and for tasks where a good response is hard to produce but easy to recognise, it is the wrong bound entirely. Comparisons ask people for the easy judgement; demonstrations ask for the hard one.

Imitation is not the thing being imitated. Gudibande et al. fine-tuned open models on outputs from a much stronger one and got a result worth remembering: human raters preferred the students, and targeted benchmarks showed no improvement at all. The models had learned the style — the confident register, the structure, the formatting — without the underlying capability.

Where it leaves us

SFT is not a rung on a ladder that gets discarded. Every pipeline in the rest of this series starts from an SFT model — it is the reference policy that PPO constrains towards, and the initialisation DPO improves. Skipping it does not work.

What it cannot do is improve past its data, and it cannot be told what to avoid. Both limits have the same shape: the signal is one-sided.

The fix is to ask the other question. Not “what should the answer be” but “which of these two is better” — cheaper to collect, available for tasks nobody can demonstrate well, and it carries a negative.

Part 3 turns a pile of comparisons into a number, which is less obvious than it sounds and introduces the failure mode that the two parts after it are both trying to contain.