SGLang

vLLM treats requests as independent and manages their memory well. SGLang starts from the observation that they usually are not.

A chat conversation resends the whole history each turn. An agent loop resends the same system prompt and tool definitions every step. A branching search shares a common prefix across every branch. In all three, most of the prefill is recomputation of something the server has already done.

RadixAttention

Cached prefixes are held in a radix tree keyed by token sequence, so lookup finds the longest matching prefix rather than requiring an exact hit.

That distinction is the point. Turn five of a conversation shares turns one to four with the previous request, and a hash of the full prompt would miss. A radix tree matches the shared span, reuses its KV blocks, and prefills only the new tokens. Eviction is LRU over tree nodes, so hot prefixes — a system prompt shared by every request — stay resident.

Constrained decoding

The other half. When output must be valid JSON matching a schema, the usual approach is to generate and retry until it parses.

Instead, compile the grammar to a finite-state machine and, at each step, mask the logits of every token that cannot legally come next. The output is valid by construction, retries disappear, and — because the mask is precomputed per state — it costs almost nothing at generation time.

import sglang as sgl

@sgl.function
def extract(s, document: str):
    s += sgl.system("Extract structured data. JSON only.")
    s += sgl.user(document)
    # The schema is enforced during decoding, not checked afterwards.
    s += sgl.assistant(sgl.gen("out", max_tokens=256, json_schema=SCHEMA))

# Or as a server, OpenAI-compatible like everything else:
#   python -m sglang.launch_server --model-path <model> --port 30000

Choosing between the two