Additive Attention

Attention was introduced without a dot product. Bahdanau et al. needed a score for each decoder-state/encoder-state pair and, rather than assuming the two should be compared by inner product, they learned the comparison.

$W_s$ and $W_h$ map the two sides into a shared $d_a$-dimensional space independently, so the decoder state and the encoder states may have different dimensions — something the dot product cannot do without an extra projection. The tanh and $v$ then reduce the pair to a scalar.

Why it lost

The nonlinearity sits between the two operands. That single fact is the whole story: it means the $n \times m$ score matrix cannot be written as a product of a $Q$ matrix and a $K$ matrix, so it cannot be one GEMM.

Luong et al. compared the two directly and found the accuracy difference small and inconsistent — dot-product scoring was competitive on their tasks and sometimes better. Given comparable quality and an order-of-magnitude difference in how well the computation maps onto matrix hardware, the outcome was not close.

What it is still good for

Reading the original formulation makes the modern one look like a choice rather than a definition. The three requirements on a scoring function — depends on both sides, normalises to a distribution, differentiable — are satisfied by both, and seeing a second solution is what stops $\softmax(QK^\top/\sqrt{d_k})$ feeling like an incantation.

It also still appears in small models where $n \cdot m$ is tiny and the extra expressivity is worth more than the GEMM: pointer networks, some retrieval rerankers, and a good deal of pre-2017 code that still works.

import torch
from torch import Tensor, nn


class AdditiveAttention(nn.Module):
    """Bahdanau scoring. d_s and d_h may differ — that is the point."""

    def __init__(self, d_s: int, d_h: int, d_attn: int) -> None:
        super().__init__()
        self.w_s = nn.Linear(d_s, d_attn, bias=False)
        self.w_h = nn.Linear(d_h, d_attn, bias=False)
        self.v = nn.Linear(d_attn, 1, bias=False)

    def forward(self, s: Tensor, h: Tensor, mask: Tensor | None = None) -> Tensor:
        # [B, N, 1, A] + [B, 1, M, A] -> [B, N, M, A]. This tensor is the cost:
        # d_attn times larger than the score matrix it collapses to.
        joint = self.w_s(s).unsqueeze(2) + self.w_h(h).unsqueeze(1)
        scores = self.v(joint.tanh()).squeeze(-1)

        if mask is not None:
            scores = scores.masked_fill(mask, torch.finfo(scores.dtype).min / 2)

        return scores.softmax(dim=-1) @ h

The unsqueeze-and-add broadcast materialises a $[B, N, M, A]$ tensor — $d_a$ times the memory of the score matrix itself. At $n = m = 4096$ and $d_a = 64$ that is 4 GB in fp32 for a single head of a single layer, which is the practical reason nobody runs this at transformer scale.