--- title: "Multi-Head Latent Attention: How DeepSeek Shrinks AI Memory" url: https://syntaxradar.com/radar/deepseek-v3-mla author: "Abdul Rehman, Lead Systems Architect and Technical Director" publishedAt: 2026-10-24T08:00:00.000Z dateModified: 2026-10-28T18:00:00.000Z category: "#LLM-MODELS" topic: "multi-head latent attention" primarySource: https://arxiv.org/abs/2405.04434 summary: "Multi-head latent attention explained in plain English: how DeepSeek's MLA shrinks the KV cache, what its papers report, and how it compares with GQA." --- # Multi-Head Latent Attention: How DeepSeek Shrinks AI Memory **Short answer:** DeepSeek introduced multi-head latent attention in DeepSeek-V2 and kept it in DeepSeek-V3. It compresses the memory a model keeps about earlier text; DeepSeek's paper reports a 93.3% smaller KV cache than its older 67B model. **Takeaway:** MLA stores one compressed summary per token instead of full keys and values for every head, which DeepSeek credits with a far smaller KV cache and cheaper long conversations. **Key figures (as reported by the source):** - KV cache vs DeepSeek 67B: 93.3% smaller - Max generation throughput: 5.76x higher - Tokens/sec on 8 H800 GPUs: 50,000+ When a chatbot answers you, it has to keep notes on everything already said in the conversation. Those notes live in expensive chip memory, and on long chats they become one of the biggest costs of running an AI model. Multi-head latent attention, usually shortened to MLA, is the design DeepSeek introduced to make those notes much smaller. DeepSeek first described MLA in its [DeepSeek-V2 paper](https://arxiv.org/abs/2405.04434) in May 2024 and kept it for [DeepSeek-V3](https://arxiv.org/abs/2412.19437), its larger open model released in December 2024. The V2 paper reports that the new model needed 93.3% less of this conversation memory than DeepSeek's earlier 67-billion-parameter model. By the end of this piece you will know what that memory is, how MLA shrinks it, what the published numbers do and do not show, and why other labs have since borrowed the idea. ### What is multi-head latent attention? Start with three terms. A **token** is a small chunk of text, often a word or part of a word. Language models read and write text one token at a time. **Attention** is the step where the model decides which earlier tokens matter for the next one. When it writes "she" in a story, attention is how it looks back and works out who "she" is. Modern models run many attention "heads" side by side, each looking for a different kind of pattern, which is why the standard version is called multi-head attention. MLA keeps that multi-head idea but changes what gets stored. Instead of saving a full, separate record for every head, it saves one compact compressed summary per token and rebuilds what each head needs from it. "Latent" is the machine learning word for that kind of hidden, compressed representation. ### Why does an AI model need so much memory to chat? To decide what comes next, attention compares the new token against every earlier one. For each earlier token the model computes two lists of numbers, called a **key** and a **value**. Roughly, the key says what that token is about and the value holds the information it contributes. Recomputing those for the whole conversation at every step would be slow, so models save them. That saved store is the **KV cache** (key-value cache). We covered how it works, and how it can leak, in our [KV cache explainer](/radar/kv-cache-explained-cloudflare-prompt-leak). The catch is that the cache grows with every token, every head and every layer of the model. The DeepSeek-V2 paper says the heavy KV cache of standard multi-head attention is a bottleneck that limits how efficiently a model can generate text. A bigger cache means fewer conversations fit on one chip at the same time, and that drives up the cost of serving each user. ![An engineer checks a row of server racks with blinking status lights in a cool data center aisle, the kind of hardware where DeepSeek MLA saves memory](https://images.unsplash.com/photo-1620712943543-bcc4688e7485?w=1600&auto=format&fit=crop&q=80) ### How does multi-head latent attention work? Picture a newsroom where 128 editors each keep their own full file on every sentence of a long story. The filing cabinets fill up fast. MLA replaces that with one short shorthand note per sentence, plus a set of fixed decoding rules that let each editor expand the note into the detail they need, when they need it. In technical terms, MLA squeezes each token's keys and values into a single small vector before caching it. The DeepSeek-V2 paper calls this "low-rank joint compression". According to [the paper's full text](https://arxiv.org/html/2405.04434v5), DeepSeek-V2 has 128 attention heads of 128 numbers each, yet the cache stores one compressed vector of 512 numbers per token per layer, plus a 64-number piece that carries position information. That position piece exists for a reason. Models need to know word order, and the usual method (called rotary position embedding) does not mix well with the compression step. DeepSeek's fix was to keep a small, separate slice for position, which the paper calls "decoupled" position embedding. There is one more trick. You might expect that rebuilding full keys and values for every head would cost a lot of extra computing. The paper explains that the decompression math can be folded into other steps of attention when the model is running, so the full keys and values do not have to be rebuilt every time. As [Sebastian Raschka's architecture comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison) puts it, MLA compresses keys and values "into a lower-dimensional space before storing them in the KV cache." The compression is learned, not bolted on later. The model is trained with MLA from the start, so it learns what to keep in those 512 numbers. ### What did DeepSeek's papers actually report? The headline figures come from the [DeepSeek-V2 paper](https://arxiv.org/abs/2405.04434), which compares the new model with DeepSeek 67B, the company's earlier model:
What was measuredDeepSeek-V2 vs DeepSeek 67B
KV cache size93.3% smaller
Maximum generation throughput5.76 times higher
Training cost42.5% lower
The paper also reports that, served on a single machine with 8 Nvidia H800 GPUs, DeepSeek-V2 generated more than 50,000 tokens per second and processed prompts at more than 100,000 tokens per second. Read these numbers with care. They compare two whole models, not MLA against standard attention in isolation. DeepSeek-V2 also uses a "mixture of experts" design, where only 21 billion of its 236 billion parameters are active for each token. For the throughput test, the paper says the team converted the model to FP8, a lower-precision number format, and compressed each KV cache element to about 6 bits on average. MLA is a large part of the memory saving, but not the only ingredient. DeepSeek-V3 then carried MLA forward at a bigger scale. Its [technical report](https://arxiv.org/abs/2412.19437) describes a model with 671 billion total parameters, 37 billion active per token, and says MLA and the expert design were "thoroughly validated in DeepSeek-V2." The report says full training took 2.788 million H800 GPU hours. The weights are public on [GitHub](https://github.com/deepseek-ai/DeepSeek-V3), which is what made the design easy for others to study. If the term is new to you, our guide to [open-weight models](/radar/open-weight-models-explained-ibm) explains what that release model means. ![A developer at a desk reviews a long document on a laptop late in the evening, the kind of long conversation where multi-head attention memory adds up](https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=1600&auto=format&fit=crop&q=85) ### Is MLA better than standard multi-head attention? Usually, shrinking memory costs some accuracy. The DeepSeek-V2 paper claims MLA avoids that trade. It says MLA "achieves superior performance compared with MHA" (multi-head attention) while needing a much smaller KV cache, based on comparison tests reported in the paper's appendix. Raschka's review reads those same tests the same way: in DeepSeek's experiments, MLA slightly outperformed standard multi-head attention. That is DeepSeek's own evidence, run on its own models and settings. Independent results on other model sizes may differ, and the company did not claim the gain holds everywhere. The honest summary: MLA is not free. It adds steps and complexity to the model, and it only pays off because the memory saved is worth more than the extra math. ### What is the difference between MLA and GQA? MLA was not the first attempt to slim the KV cache. A popular earlier approach is **grouped-query attention**, or GQA, used in many open models. GQA puts the attention heads into groups and lets each group share one set of keys and values. Fewer distinct keys and values means a smaller cache. The difference is in how they save space. GQA shares; MLA compresses. GQA throws away some per-head detail by making heads share. MLA keeps a single dense summary that every head can expand into its own version. The DeepSeek-V2 paper puts a number on it. It says MLA's cache is about the same size as GQA with only 2.25 groups, a very aggressive setting, yet it reports stronger results than full multi-head attention. In DeepSeek's tests, GQA scored below standard attention, while MLA scored above it, according to Raschka's summary. The trade-off is engineering effort. Raschka notes that MLA is more complicated to implement than GQA, which helps explain why GQA remains common in smaller models. ### What to watch next The clearest sign that MLA works is that others now copy it. According to Raschka's comparison, Moonshot AI's Kimi K2 uses the DeepSeek-V3 architecture, including MLA, with more experts and fewer attention heads. DeepSeek keeps building on it too. Its [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556), published in December 2025, adds DeepSeek Sparse Attention, which picks only the most relevant earlier tokens to look at. The paper says that new layer is built on top of MLA rather than replacing it. For everyday users the effect is indirect. Smaller memory needs per conversation mean one chip can serve more chats at once, which lowers the cost of each one. Watch for more open models adopting MLA, for serving software adding faster support for it, and for independent tests that check whether its quality claims hold outside DeepSeek's own lab. ## Frequently asked questions ### What is multi-head latent attention in simple terms? It is a way for an AI model to keep shorter notes about the conversation so far. Instead of storing full keys and values for every attention head, it stores one compressed summary per token and expands it when needed. DeepSeek introduced it in DeepSeek-V2 to cut the memory needed during text generation. ### How much memory does DeepSeek MLA save? The DeepSeek-V2 paper reports a 93.3% smaller KV cache than DeepSeek 67B, its earlier model. That compares two whole models, not MLA alone. DeepSeek-V2 also uses a mixture-of-experts design, and its serving setup used FP8 weights and roughly 6-bit KV cache compression, so several choices contribute. ### Does MLA make AI models less accurate? DeepSeek says no. Its V2 paper reports that MLA performed better than standard multi-head attention in its own comparison tests while using a much smaller cache. Those tests were run by DeepSeek on its own models, so independent checks on other model sizes and settings are still worth watching. ### Which AI models use DeepSeek MLA? DeepSeek-V2, DeepSeek-V3 and DeepSeek-R1 use it, and DeepSeek-V3.2 builds its sparse attention on top of it. According to Sebastian Raschka's architecture comparison, Moonshot AI's Kimi K2 also uses the DeepSeek-V3 design with MLA, adjusted to have more experts and fewer attention heads. ## Source Based on DeepSeek's V2, V3 and V3.2 papers. https://arxiv.org/abs/2405.04434 --- Published on SyntaxRadar (https://syntaxradar.com). Canonical URL: https://syntaxradar.com/radar/deepseek-v3-mla. Last modified 2026-10-28T18:00:00.000Z.