Next-Token Prediction

That is not a simplification. Everything a base model knows was obtained by minimising that quantity over a large corpus, and the exponential of it is perplexity.

Two properties make it the objective that scaled. It needs no labels — the text is its own supervision, so the training set is as large as the internet. And with causal masking every position is a training example computed in parallel, so one forward pass over a 4096-token document yields 4096 predictions rather than one.

Teacher forcing

During training the model conditions on the true prefix at every position. It never sees its own output.

At generation time the model conditions on what it produced, which is drawn from a distribution it was never trained to handle. Once a token appears that no plausible human prefix would contain, the model is off the manifold it learned, and errors compound. The classical name is exposure bias.

Scheduled sampling — mixing in the model’s own predictions during training — was the obvious fix and is not used. It biases the gradient, because the mixed prefix is not drawn from the data distribution, and at scale the problem it addresses largely recedes: a model good enough to stay near the data manifold does not accumulate the error. What remains of it is visible as degeneration under greedy decoding, which is handled at the sampler instead.

What the loss weights

Averaging over tokens means every token counts equally: the closing bracket of a code block, the second half of a common word, and the one word in a paragraph that carries its meaning. Most of the loss, by mass, is on tokens that are nearly free to predict.

The gradient with respect to the logits is the predicted distribution minus a one-hot vector — bounded, cheap, and never saturating. A great deal of the transformer’s trainability comes from this pairing of softmax with cross-entropy rather than from the architecture.

import torch
from torch import Tensor
from torch.nn import functional as F


def loss(logits: Tensor, tokens: Tensor, ignore: int = -100) -> Tensor:
    """logits: [B, T, V] over positions 0..T-1. tokens: [B, T]."""
    # The shift is the objective: position t predicts token t+1.
    pred = logits[:, :-1].reshape(-1, logits.size(-1))
    target = tokens[:, 1:].reshape(-1)

    # fp32 for the softmax denominator: in bf16 the log-sum-exp over 128k
    # vocabulary entries loses enough mantissa to bias the loss.
    return F.cross_entropy(pred.float(), target, ignore_index=ignore)

The off-by-one in that shift is the most common bug in a from-scratch implementation, and it does not crash. A model trained to predict the current token learns the identity and reaches a suspiciously low loss, which is the tell.

The one active line of work on the objective itself. Gloeckle et al. add $n$ output heads predicting tokens $t{+}1$ through $t{+}n$ from the same trunk, at negligible extra cost since the trunk dominates.

Two results follow. Larger models improve on generation benchmarks — the extra heads act as a denser training signal, forcing the representation at position $t$ to carry more than the immediate next token. And the auxiliary heads give speculative decoding a draft model for free, which is a straightforward two- to three-fold inference speed-up. DeepSeek-V3 ships an MTP objective for that reason.