GELU

Not a threshold but a probability: multiply the input by the chance that a standard normal falls below it. The result looks like ReLU with the corner rounded off, and the rounding is the point.

[ffn][core]Commonly used

Standing

Commonly usedEstablished and frequently the right choice, but competing with live alternatives rather than having settled the question.

In BERT, GPT-2, GPT-3 and every ViT. New models use SiLU inside a gate instead, and the difference between them is not measurable.

judged as of 2026-09 · what the labels mean

Theory

ReLU makes a hard decision: keep the input or discard it. GELU makes a soft one — scale the input by how likely it is to be worth keeping, where “likely” is read off a standard normal.

\mathrm{GELU}(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[1 + \mathrm{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right]

The motivation in the original paper is a stochastic regulariser: dropout multiplies by a Bernoulli mask, and if you make the keep-probability depend on the input’s magnitude and then take the expectation, $x\Phi(x)$ is what falls out. Whether that story is load-bearing is doubtful. What is not doubtful is that the function has a small negative lobe and a continuous derivative, and both help.

The family

Three functions are in circulation and they are near-identical over the range activations actually occupy.

Name	Form	Where
GELU	$x\,\Phi(x)$	BERT, GPT-2/3, ViT
SiLU / Swish	$x\,\sigma(x)$	Llama, Mistral (inside a gate)
GELU-tanh	$x\,\Phi_{\tanh}(x)$	the fast approximation

SiLU replaces the Gaussian CDF with a logistic one, which is the same S-curve with slightly heavier tails. Ramachandran et al. found it by automated search, which is a pleasing result about how much of activation design is taste: a search over thousands of candidates rediscovered the function someone had already derived from a dropout argument.

\mathrm{GELU}_{\tanh}(x) = \frac{x}{2}\left[1 + \tanh\!\left(\sqrt{\tfrac{2}{\pi}}\,\bigl(x + 0.044715\,x^3\bigr)\right)\right]

That constant is a curve fit, not a derivation. The approximation exists because erf was slow on the hardware of 2018; it is no longer slow, and the approximation persists because checkpoints were trained against it.

Implementation

python · torch

import math

import torch
from torch import Tensor


def gelu_exact(x: Tensor) -> Tensor:
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))


def gelu_tanh(x: Tensor) -> Tensor:
    """GPT-2's form. Use this iff the checkpoint was trained with it."""
    inner = math.sqrt(2.0 / math.pi) * (x + 0.044715 * x.pow(3))
    return x * 0.5 * (1.0 + torch.tanh(inner))


# torch spells the choice as a flag, which is the right way to carry it:
#   nn.GELU()                     -> exact
#   nn.GELU(approximate="tanh")   -> GPT-2

Does the choice matter

Barely, and it is worth saying so plainly. Shazeer’s GLU-variants paper trained the whole matrix — ReLU, GELU, Swish, each with and without a gate — and the gated versions beat the ungated ones by a clear margin while the activations within each group were separated by noise.

The conclusion the field drew, correctly, is that gating is the design decision and the activation inside it is a detail. Llama uses SiLU because SwiGLU is what Shazeer named; had he named GEGLU, the models would use GELU and nothing else would differ.

erf-based

exact

tanh approximation

≈ 1e−3 error

sigmoid (SiLU)

different function

Cost of the exact form

swiglu feed-forward-networks squared-relu activation-sparsity rmsnorm

References

[1]Hendrycks & Gimpel — Gaussian Error Linear Units (GELUs) (2016)arXiv:1606.08415

[2]Ramachandran et al. — Searching for Activation Functions (2017)arXiv:1710.05941

[3]Shazeer — GLU Variants Improve Transformer (2020)arXiv:2002.05202