Supervised Fine-Tuning

A base model completes text. It does not answer questions, because a question in its training distribution was as likely to be followed by more questions as by an answer. SFT changes the distribution, not the objective.

Data, not gradient steps

LIMA’s result is the one worth internalising: 1,000 carefully curated examples outperformed 52,000 scraped ones. The claim is that the model already has the capability from pretraining and SFT is selecting a response format — a small, consistent, high-quality set specifies that format better than a large inconsistent one.

The practical consequences are all about the data:

• Deduplicate aggressively. A repeated example is a repeated gradient.
• Keep the format uniform. Two chat templates in one dataset teach neither.
• Cap the epochs at 2–3. Beyond that the model memorises the demonstrations and its held-out loss rises while its training loss keeps falling.

A rising validation loss during SFT is not automatically a problem. The objective is not to predict held-out demonstrations well; it is to produce useful completions, and those two diverge early.

import torch
from torch import Tensor

IGNORE = -100   # cross_entropy skips these positions


def build_example(tokenizer, prompt: str, completion: str) -> dict[str, Tensor]:
    prompt_ids = tokenizer(prompt, add_special_tokens=False).input_ids
    completion_ids = tokenizer(
        completion + tokenizer.eos_token, add_special_tokens=False
    ).input_ids

    input_ids = prompt_ids + completion_ids
    labels = [IGNORE] * len(prompt_ids) + completion_ids

    return {
        "input_ids": torch.tensor(input_ids),
        "labels": torch.tensor(labels),
    }

The EOS token belongs inside the supervised span. Leave it out and the model is never taught to stop, which shows up as completions that run on until the length cap. When packing several examples into one sequence for throughput, either reset the attention mask at every document boundary or accept that the model can attend across unrelated examples — most implementations quietly do the latter.