Sliding-Window Attention

Each token attends only to the w tokens preceding it, making cost linear in sequence length and the effective receptive field L·w after L layers.

[attention][sparse][stub]

Status

This entry is a stub. The maths has not been rederived by hand, or the implementation has not been run on real tensors, so it is recorded here as an open question rather than an answer.

Read what follows as a pointer to the sources, not as a settled account.

Notes

Masking every query to a band of $w$ preceding keys makes the cost $O(n \cdot w \cdot d)$ rather than $O(n^2 d)$ , and stacking $L$ such layers gives an effective receptive field of $L \cdot w$ — information propagates, but only at one window per layer.

What is not yet settled here, and the reason this is a stub, is how the band interacts with the position scheme and with attention sinks: the first few tokens appear to matter far more than the band alone would predict.

scaled-dot-product-attention flash-attention

References

[1]Beltagy et al. — Longformer: The Long-Document Transformer (2020)arXiv:2004.05150

[2]Child et al. — Generating Long Sequences with Sparse Transformers (2019)arXiv:1904.10509