---
title: "SGLang vs vLLM: How Prefix Caching Cuts Wait Times"
url: https://syntaxradar.com/radar/radixattention-dynamic-prefix-caching-sglang-vllm-benchmarks
author: "Abdul Rehman, Lead Systems Architect and Technical Director"
publishedAt: 2026-10-25T07:40:00.000Z
dateModified: 2026-10-29T10:33:58.900Z
category: "#SYSTEMS-ARCH"
topic: "sglang vs vllm"
primarySource: https://arxiv.org/abs/2312.07104
summary: "SGLang vs vLLM explained: how RadixAttention and prefix caching skip repeated prompt work, what the benchmarks report, and how to test it on your traffic."
---


# SGLang vs vLLM: How Prefix Caching Cuts Wait Times

**Short answer:** SGLang's RadixAttention and vLLM's automatic prefix caching both reuse stored work on repeated prompt openings. Published speedups come mostly from shared-prefix workloads, and newer comparisons conflict.

**Takeaway:** Prefix caching helps most when many requests share a long opening, and published speedups come mostly from such workloads.

**Key figures (as reported by the source):**
- SGLang paper: peak throughput gain: up to 6.4x
- LMSYS 2024 blog: peak throughput gain: up to 5x
- vLLM prefix cache eviction: Least recently used

SGLang vs vLLM comes down to one question: how much of your prompt do your requests have in common? SGLang is built around a prefix cache called RadixAttention, and vLLM offers a similar feature called automatic prefix caching. Where requests share long openings, both skip repeated work. Where they share nothing, neither has much to skip. The headline speedups you see online come mostly from the first situation.

Both tools are inference engines: the software that sits between a trained language model and the people sending it requests. Prefix caching is the trick of remembering work done on the start of a prompt so it is not repeated.

### What is RadixAttention in SGLang?

RadixAttention is SGLang's method for reusing the model's working memory across requests. It was introduced by the LMSYS team in a [January 2024 announcement](https://lmsys.org/blog/2024-01-17-sglang/) and described in the [SGLang paper](https://arxiv.org/abs/2312.07104) by a twelve-author team that includes Lianmin Zheng, Ion Stoica and Joseph E. Gonzalez.

When a model reads a prompt, it stores intermediate results for every token in what is called the KV cache (key-value cache), so it does not have to reread earlier words while writing a reply. Our [explainer on the KV cache](/radar/kv-cache-explained-cloudflare-prompt-leak) covers the basics. Normally that memory is thrown away when a request finishes.

According to LMSYS, RadixAttention instead keeps the KV cache for both prompts and generated answers in a radix tree. A radix tree is a lookup structure that stores sequences of items along shared branches, the way a file system shares folder paths. Here the items are tokens (word fragments), so two prompts that begin the same way share the same branch and the same stored memory.

### How does prefix caching work, step by step?

Reading a prompt is called prefill. It is the first phase of every request, and it must finish before the first word of the reply appears. The delay between sending a request and seeing that first word is the time to first token.

If a new request begins with text the engine has already processed, prefill for that part can be skipped. Only the new tail needs computing. The longer the shared opening, the more work is saved.

LMSYS describes three supporting pieces. The tree itself is kept on the CPU, while the stored tensors stay on the GPU. When GPU memory runs short, a least-recently-used policy evicts the oldest leaf nodes first. A cache-aware scheduler also tries to order incoming requests so that more of them find a match in the tree.

The same post says the design works with continuous batching and paged attention, two standard techniques for keeping GPUs busy, and that the authors saw no noticeable overhead when there were no cache hits.

![A technician in a data center aisle checking rows of server racks lit by blue status lights](https://images.unsplash.com/photo-1544197150-b99a580bb7a8?w=1600&auto=format&fit=crop&q=80)

### Is vLLM's prefix caching different from RadixAttention?

Both engines do the same job, but they track shared text differently. SGLang follows branches of a tree, while vLLM looks up fingerprints. vLLM's [design documentation](https://docs.vllm.ai/en/latest/design/prefix_caching/) says it hashes each block of the KV cache. A hash is a short fingerprint computed from content. The fingerprint of a block combines the fingerprint of the block before it, the tokens inside it, and extras such as adapter IDs or multimodal inputs. Only full blocks are cached, and eviction is least-recently-used as well.

A new request is checked block by block against those fingerprints, so a match is found by lookup. The docs describe the feature as letting a query that shares a prefix with a cached one reuse that cache. Their [feature page](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html) lists multi-turn chat and repeated questions about one long document as the cases where it helps.

Because only full blocks are cached, a prefix that ends partway through a block cannot be reused beyond the last full block. How much that matters in practice depends on block size and the workload.

The same feature page says automatic prefix caching speeds up prefill only. It does nothing for long generated answers, and it does nothing for new queries that share no prefix with anything cached.

### SGLang vs vLLM: what do the benchmarks actually show?

The original SGLang paper reports "up to 6.4x higher throughput" than state-of-the-art inference systems across several tasks, including agent control, retrieval-augmented generation, JSON output and multi-turn chat. The LMSYS blog post, which tested earlier, says "up to 5 times higher throughput" against vLLM v0.2.5, Guidance and Hugging Face TGI, on Llama-7B using one NVIDIA A10G GPU and on Mixtral-8x7B using eight.

Two cautions apply. First, those baselines are from late 2023 and early 2024. vLLM has since added automatic prefix caching, so the comparison no longer matches what a reader faces today. Second, "up to" means the best case among the workloads tested.

Newer comparisons are mostly blog posts, and their numbers conflict. One 2026 post on DEV Community reports median time to first token of 380 ms on vLLM against 85 ms on SGLang for a shared-prefix test. Another post from Spheron reports 1,920 tokens per second for SGLang against 1,850 for vLLM at 50 concurrent requests on an H100, which is close to a tie. These are single-author tests with their own settings and software versions.

A [SwiftKV paper](https://arxiv.org/pdf/2410.03960) on arXiv mentions SGLang giving 1.4x to 1.8x higher throughput than vLLM on Llama-3.1-8B. That is a smaller gap than the original claims and was measured in a different context.

![A developer comparing two sets of performance numbers on dual monitors in a quiet office](https://images.unsplash.com/photo-1526374965328-7f61d4dc18c5?w=1600&auto=format&fit=crop&q=80)

### Which one should you choose?

Prefix caching pays off when many requests begin with the same material. Examples include a long system prompt reused by every user, a tool list that an agent resends on each step, or a document that many questions refer to. In those cases, skipping repeated prefill cuts waiting time before the first word.

It pays off less when prompts are mostly unique, or when the reply is very long. The vLLM docs say outright that prefix caching does not speed up the generation phase. If your cost is dominated by long answers, reusing prompt memory moves little.

Features beyond caching also matter when picking an engine, such as model support, structured output, hardware coverage and how easily it fits your existing deployment.

To compare them, take a sample of real traffic, run it through both engines on the same hardware with prefix caching switched on in vLLM, and compare time to first token and throughput at the concurrency you expect. The vLLM page says to set `enable_prefix_caching=True` to turn the feature on, and does not state the default, so check your version.

### What to watch next

Neutral head-to-head tests on current versions are missing. Today most comparisons come from vendors, hosting companies and individual bloggers, rather than from a neutral lab using matched settings.

Memory pressure is another open area. Both rely on least-recently-used eviction, which is simple but can discard a prefix that a busy service is about to need again. The SGLang team's cache-aware scheduler is one answer to that problem, and vLLM's documentation describes its own eviction rules in detail.

Research on newer model designs is a further area. A paper on arXiv titled [Marconi: Prefix Caching for the Era of Hybrid LLMs](https://arxiv.org/pdf/2411.19379) suggests that prefix caching is being rethought for models that are not plain transformers. If model designs keep changing, caches built around one design may need rework. Our piece on [DeepSeek's multi-head latent attention](/radar/deepseek-v3-mla) covers another example of how a change in memory design affects serving costs.

## Frequently asked questions

### What is RadixAttention?
RadixAttention is SGLang's way of reusing the model's stored memory (the KV cache) across requests. According to LMSYS, it keeps the cache for prompts and generated answers in a radix tree, so requests that begin with the same tokens share the same stored work instead of recomputing it from scratch.

### Does vLLM have prefix caching?
Yes. vLLM documents automatic prefix caching, which lets a new query reuse the cache of an earlier query that shares its opening. It works by hashing blocks of the cache. The feature page says to set enable_prefix_caching=True to turn it on and does not state the default, so check your version.

### Is SGLang faster than vLLM?
It depends on the workload. The SGLang paper reports up to 6.4x higher throughput against older baselines, but newer blog comparisons conflict, with one showing a near tie on throughput. Prefix caching helps most when requests share long openings, so the only reliable answer comes from testing your own prompts.

### When does prefix caching not help?
The vLLM documentation says it only speeds up the prefill phase, not token generation. It does not help with long generated answers, and it does not help when a new query shares no opening with anything already cached. Mostly unique prompts see little benefit.

## Source
Based on the SGLang paper and vLLM docs. https://arxiv.org/abs/2312.07104

---
Published on SyntaxRadar (https://syntaxradar.com). Canonical URL: https://syntaxradar.com/radar/radixattention-dynamic-prefix-caching-sglang-vllm-benchmarks. Last modified 2026-10-29T10:33:58.900Z.