Inference

Serving is a memory-bandwidth problem wearing a compute problem’s clothes. Kernels that avoid materialising intermediates, cache layouts that avoid re-reading them, and quantisation schemes that shrink what must be read at all.

Eight entries across five branches. One is a stub and is marked as such — read them as open questions rather than answers.

KV cache

kv-cache/ ›

the memory that decoding is really bound by

04.1

Multi-Query Attentionstale

Single shared K/V head across query heads to shrink the decode cache.

O(n·d)04.2

Paged KV Cachestubstandard

Virtual-memory paging applied to the attention cache.

O(1)04.3

Prefix Cachingstandard

Reuse KV blocks across requests that share a prompt prefix.

O(1)

Decoding

decoding/ ›

turning a distribution into a token, and the shortcuts around it

04.4

Sampling Strategiesstandard

Temperature, top-k, nucleus and min-p as successive truncations of the tail.

O(V log V)04.5

Speculative Decodingstandard

Draft model proposes tokens; target model verifies in one pass.

O(γ·n)

Quantisation

quantisation/ ›

fewer bits per weight, and where the error goes

04.6

GPTQcommon

Layer-wise second-order weight quantisation with Hessian-guided rounding.

O(d³)

Kernels

kernels/ ›

IO-aware implementations of operations that already existed

04.7

FlashAttentionstandard

Tiled, IO-aware attention kernel; recomputes softmax statistics online.

O(n²·d)

Serving

serving/ ›

scheduling many requests against one set of weights

04.8

Continuous Batchingstandard

Iteration-level scheduling; requests join and leave the batch mid-flight.

O(1)