LSTM and GRU

The vanishing gradient is a product of Jacobians. The LSTM’s answer is to add a path along which that Jacobian is approximately the identity.

No matrix multiplies $c_{t-1}$. It is scaled elementwise by a gate and something is added. If $f_t \approx 1$ the derivative $\partial c_t / \partial c_{t-1}$ is approximately 1, and the gradient traverses a hundred steps undiminished — the constant error carousel, in the original paper’s phrase.

Two states, not one. $c$ is the memory and is only ever gated and added to; $h$ is the exposed view of it and is what the next layer and the next step see. Keeping them separate is what allows the model to hold something without acting on it.

GRU

Cho et al.’s simplification merges the cell and hidden states, and ties the input and forget gates so that what is admitted is exactly what is displaced.

Three gates instead of four, about 25% fewer parameters, and — across every careful comparison, including Greff et al.’s eight-variant study — no consistent difference in quality. Which variant to use is a matter of what the framework defaults to.

What actually killed it

Not quality. LSTMs were state of the art in translation until 2017 and their long-range behaviour is respectable.

They were killed by eq. 1’s shape. $h_t$ depends on $h_{t-1}$, so training cannot parallelise over the sequence — a length-1024 sequence is 1024 dependent kernel launches. Attention computes all positions at once, and on hardware that rewards parallelism above all else, the quadratic cost was worth paying.

import torch
from torch import Tensor, nn


class LSTMCell(nn.Module):
    def __init__(self, dim: int, hidden: int):
        super().__init__()
        # One matmul for all four gates, then split — four small matmuls
        # would be four kernel launches per step, and there are T of those.
        self.w = nn.Linear(dim + hidden, 4 * hidden)
        self.hidden = hidden

        # Forget-gate bias to 1: start out remembering, learn to forget.
        with torch.no_grad():
            self.w.bias[hidden : 2 * hidden].fill_(1.0)

    def forward(self, x: Tensor, state: tuple[Tensor, Tensor]) -> tuple[Tensor, Tensor]:
        h, c = state
        i, f, g, o = self.w(torch.cat([x, h], dim=-1)).chunk(4, dim=-1)

        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
        h = torch.sigmoid(o) * torch.tanh(c)
        return h, c

The fused gate projection is the standard optimisation and it is why cuDNN’s LSTM is several times faster than a hand-written loop: it fuses across gates, across layers, and across the batch, leaving only the unavoidable dependency along $t$.

The architecture is gone; the gate is everywhere. SwiGLU is a multiplicative gate on a feed-forward block. Mamba’s selection mechanism is an input-dependent forget gate under another name. RWKV writes its decay as exactly that. The 1997 insight — that a network should learn what to keep rather than being forced to transform everything — outlived the recurrence it was invented to rescue.