--- title: "Mixture of Depths: How Google DeepMind Lets AI Skip Layers" url: https://syntaxradar.com/radar/mixture-of-depths-dynamic-compute-routing-deepmind-mod-benchmarks author: "Abdul Rehman, Lead Systems Architect and Technical Director" publishedAt: 2026-10-25T07:55:00.000Z dateModified: 2026-10-28T18:00:00.000Z category: "#LLM-MODELS" topic: "mixture of depths" primarySource: https://arxiv.org/abs/2404.02258 summary: "Mixture of depths lets a transformer skip layers for easy tokens. What Google DeepMind's paper measured, how it differs from MoE, and what it did not prove." --- # Mixture of Depths: How Google DeepMind Lets AI Skip Layers **Short answer:** Google DeepMind's 2024 mixture-of-depths paper trained transformers whose routers let most tokens skip some layers. Models up to 3B parameters matched or beat standard ones at equal training compute while using fewer FLOPs per forward pass. **Takeaway:** DeepMind's MoD research shows, at research scale, that a model can learn which tokens deserve full processing and save compute without losing quality, though big-model and real-world speed results are still missing. **Key figures (as reported by the source):** - Tokens processed per routed layer: 12.5% (256 of 2,048) - Largest model tested: About 3B parameters - Sampling step speed: Upwards of 50% faster Every time an AI chatbot writes a word, it pushes that word through the same long stack of calculations, whether the word is "the" or the answer to a tricky math question. Mixture of depths (MoD) is a technique from Google DeepMind, published in April 2024, that lets a language model decide which words deserve the full treatment and which can skip some of the work. The idea matters because that work costs money, electricity and waiting time. If a model can spend less effort on easy parts of a sentence without getting worse, the same hardware can do more. By the end of this piece you will know what MoD does, what the researchers actually measured, how it differs from the better-known "mixture of experts" approach, and which claims about it go further than the evidence. ### What problem was DeepMind trying to solve? A few terms first. A **transformer** is the design behind almost every modern chatbot, including ChatGPT, Gemini and Claude. It reads and writes text as **tokens**, which are small chunks of text: often a whole word, sometimes part of one, sometimes a comma. A transformer is built from a stack of **layers** (the paper calls them blocks). Each layer looks at every token, compares it with the tokens around it, and refines the model's understanding of it. Text passes through the stack one layer after another, like a document moving down a line of editors. **Compute** is the amount of arithmetic a computer does. Researchers count it in FLOPs, short for floating-point operations, which are individual multiplications and additions. More FLOPs means more chip time, more power and a bigger bill. The paper, titled [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://arxiv.org/abs/2404.02258), opens with the core complaint in one line: "Transformer-based language models spread FLOPs uniformly across input sequences." In plain English, a standard model gives every token exactly the same amount of effort at every layer. A full stop gets as much arithmetic as the key word in a hard question. People do not read that way. We skim the filler and slow down on the parts that carry meaning. The authors, David Raposo, Adam Santoro and colleagues at Google DeepMind (one co-author is also at McGill University and Mila), asked whether a model could learn to do something similar. ### What is mixture of depths and how does it work? MoD adds a small decision-maker, called a **router**, in front of a layer. For each token, the router produces a score that says roughly "how much does this token need this layer right now?" Only the highest-scoring tokens go through the layer's heavy calculations. The rest take a shortcut: they pass straight to the next layer unchanged. In technical terms they ride the "residual connection", a bypass that every transformer layer already has, so nothing new needs to be built for the skip. Here is the part that makes it practical. The model does not let any number of tokens through. It sets a fixed **capacity** in advance. In the setup the paper found worked best, a routed layer processes only 12.5% of the tokens, which is 256 out of a 2,048-token passage, and the authors applied this to every other layer. The layers in between still process every token. That fixed cap is the clever engineering choice. Chips like GPUs run fastest when they know the size of the job ahead of time. Because the number of selected tokens is always the same, the computer always does a known amount of work, even though which tokens get picked changes from passage to passage. The router is not a hand-written rule like "skip punctuation". It is trained along with the rest of the model and learns its own sense of which tokens matter. The paper observes that some tokens engage with almost every layer, while others "route around blocks whenever possible". The researchers also tried picking tokens at random, and that version performed "drastically worse", which tells you the learned choice is doing real work. Different tokens end up passing through different numbers of layers, so the model is effectively a mix of shallow and deep paths, which is where the name comes from. ![A software engineer at a standing desk in a quiet office at dusk, reading a printed mixture-of-depths research paper with a laptop open beside her](https://images.unsplash.com/photo-1618005182384-a83a8bd57fbe?w=1600&auto=format&fit=crop&q=80) ### What did the paper actually show? The team trained models ranging from 60 million to 3 billion parameters (parameters are the adjustable numbers a model learns; today's leading chatbots are far larger). They compared MoD models against standard transformers given the same total training budget, testing at three budgets: 6e18, 2e19 and 1e20 FLOPs. Three results stand out, all from the [paper's full text](https://arxiv.org/html/2404.02258v1): - **Equal or better quality for the same training spend.** The authors report that MoD "improves upon vanilla transformers by as much as 1.5%" on the training objective, a measure of how well the model predicts the next token, when both get the same training FLOPs. - **Less work per step.** Because most tokens skip half the routed layers, the models need, in the abstract's words, "a fraction of the FLOPs per forward pass". A forward pass is one trip of the text through the model. - **Faster steps.** The paper says the models "can be upwards of 50% faster to step during post-training sampling", meaning when the trained model generates text. It also reports training steps that are upwards of 60% faster on equivalent hardware. There was a catch the authors had to fix. Picking the "top" tokens means comparing every token in a passage, including ones later in the sentence. When a chatbot writes text one token at a time, those later tokens do not exist yet. Their solution was a second, tiny predictor that learns to guess the router's decision using only the current token. The paper reports this guess is upwards of 97% accurate and that switching to it caused little loss in quality. ### How is mixture of depths different from mixture of experts? You may have heard of **mixture of experts**, often shortened to MoE. It is the design behind several large open models, including the one in our [DeepSeek-V3 explainer](/radar/deepseek-v3-mla). The two ideas share a family resemblance, and the paper says so directly: MoD borrows MoE's routing logic, but "MoD deploys a single expert which can be dynamically skipped." The difference is what the router chooses.
Mixture of expertsMoD
What the router decidesWhich of several specialist sub-networks handles a tokenWhether a token goes through a layer at all
Does every token get processed at every layer?Yes, by some expertNo, many tokens skip routed layers
Main goalA bigger model without a matching rise in compute per tokenLess compute spent on tokens that do not need it
Put simply, mixture of experts asks "who should handle this word?" while MoD asks "does this word need handling here at all?" The two are not rivals. The paper tested a combination it called MoDE and reports that one way of combining them worked "distinctly better" than simply trimming expert capacity in a standard MoE model, though it gives these results as charts rather than headline numbers. MoD also differs from an older idea called early exit, where a token stops at some layer and never goes deeper. With MoD, the authors note, "a token can skip middle layers" and still be processed by later ones. ![A data center technician walking between tall rows of server racks with blinking status lights, the kind of hardware that runs transformer models](https://images.unsplash.com/photo-1634017839464-5c339ebe3cb4?w=1600&auto=format&fit=crop&q=80) ### What the paper did not prove It is easy to oversell this research, and earlier coverage (including our own first version of this article) did. Here is what the evidence does and does not support. **It does not show a model running twice as fast.** The "upwards of 50%" figure describes how quickly a sampling step can run, and the paper is careful about how it measured it. In its scaling charts, the authors say they "depict normalized FLOPs per forward pass rather than wall-clock step time per se." Fewer calculations usually means faster responses, but real-world speed also depends on memory, software and hardware. **It was tested on small models.** The largest model in the study had about 3 billion parameters. That is useful for research but small next to the commercial systems people use daily. Whether the gains hold at much larger sizes is an open question the paper does not answer. **It reports training quality, not real tasks.** The 1.5% improvement is on the model's own prediction objective. The paper does not report results on everyday benchmarks such as question answering or coding tests. **Memory savings were not measured.** Tokens that skip a layer also skip storing some of that layer's working notes, the cached data explained in our [KV cache guide](/radar/kv-cache-explained-cloudflare-prompt-leak). The authors mention this possible benefit but write, "We did not study this extensively." **There is no public sign it ships in a product.** The paper does not say whether Google uses MoD in Gemini or any other service, and we found no public confirmation that a major commercial model uses it. None of this makes the work less interesting. It makes it what it is: a careful research result showing that a model can learn where to spend its effort, at a modest scale. ### What to watch next The bigger question is whether the idea can be added to existing models cheaply, since few companies want to retrain from scratch. A later paper called [Router-Tuning](https://arxiv.org/abs/2410.13184), by researchers including Shwai He and Tao Ge, tries exactly that by training only the routers on a small dataset. Its authors report a 21% speedup with only a 0.2% drop in performance, a smaller gain than the original paper's best case but on models that already existed. Watch the name, too. In March 2026, a separate team published [Mixture-of-Depths Attention](https://arxiv.org/abs/2603.15619), or MoDA, which lets each layer look back at information from earlier layers. Despite the similar name, it solves a different problem, so search results that say "mixture of depths attention" may not be about the 2024 token-skipping method. For readers who just want the takeaway: AI models have long spent the same effort on every scrap of text. MoD showed, at research scale, that a model can learn to skip the easy parts and lose nothing on its training goal. If that holds at larger sizes, it is one more lever, alongside cheaper hardware and smarter caching, for bringing down the cost of running AI. ## Frequently asked questions ### What is mixture of depths in simple terms? It is a method from Google DeepMind that lets a language model skip some of its own layers for some tokens. A small trained router picks which tokens get full processing at a layer, and the rest pass straight through. In the paper's best setup, routed layers handled 12.5% of tokens, and this was applied to every other layer. ### Does MoD make AI twice as fast? No, the paper does not claim that. It reports models that can be upwards of 50% faster to step when generating text and upwards of 60% faster to step in training. The authors note their scaling charts show normalized FLOPs per forward pass rather than measured wall-clock time, so real speed in a product could differ. ### Does skipping layers make the model less accurate? In the paper's tests, no. With the same training budget, mixture-of-depths models matched standard transformers and improved on them by as much as 1.5% on the training objective. Skipping at random performed far worse, so the gain depends on the learned router. The paper did not report results on everyday task benchmarks. ### Is MoD used in ChatGPT or Gemini? There is no public confirmation of that. The 2024 paper tested models from 60 million to 3 billion parameters and does not say whether Google uses the method in Gemini. Later research, such as Router-Tuning, explores adding it to existing models more cheaply, which is the step that would make product use more likely. ## Source Based on Google DeepMind's arXiv paper. https://arxiv.org/abs/2404.02258 --- Published on SyntaxRadar (https://syntaxradar.com). Canonical URL: https://syntaxradar.com/radar/mixture-of-depths-dynamic-compute-routing-deepmind-mod-benchmarks. Last modified 2026-10-28T18:00:00.000Z.