Mixture of Depths: How Google DeepMind Lets AI Skip Layers

Every time an AI chatbot writes a word, it pushes that word through the same long stack of calculations, whether the word is "the" or the answer to a tricky math question. Mixture of depths (MoD) is a technique from Google DeepMind, published in April 2024, that lets a language model decide which words deserve the full treatment and which can skip some of the work.

The idea matters because that work costs money, electricity and waiting time. If a model can spend less effort on easy parts of a sentence without getting worse, the same hardware can do more.

By the end of this piece you will know what MoD does, what the researchers actually measured, how it differs from the better-known "mixture of experts" approach, and which claims about it go further than the evidence.

What problem was DeepMind trying to solve?

A few terms first. A transformer is the design behind almost every modern chatbot, including ChatGPT, Gemini and Claude. It reads and writes text as tokens, which are small chunks of text: often a whole word, sometimes part of one, sometimes a comma.

A transformer is built from a stack of layers (the paper calls them blocks). Each layer looks at every token, compares it with the tokens around it, and refines the model's understanding of it. Text passes through the stack one layer after another, like a document moving down a line of editors.

Compute is the amount of arithmetic a computer does. Researchers count it in FLOPs, short for floating-point operations, which are individual multiplications and additions. More FLOPs means more chip time, more power and a bigger bill.

The paper, titled Mixture-of-Depths: Dynamically allocating compute in transformer-based language models, opens with the core complaint in one line: "Transformer-based language models spread FLOPs uniformly across input sequences." In plain English, a standard model gives every token exactly the same amount of effort at every layer. A full stop gets as much arithmetic as the key word in a hard question.

People do not read that way. We skim the filler and slow down on the parts that carry meaning. The authors, David Raposo, Adam Santoro and colleagues at Google DeepMind (one co-author is also at McGill University and Mila), asked whether a model could learn to do something similar.

What is mixture of depths and how does it work?

MoD adds a small decision-maker, called a router, in front of a layer. For each token, the router produces a score that says roughly "how much does this token need this layer right now?"

Only the highest-scoring tokens go through the layer's heavy calculations. The rest take a shortcut: they pass straight to the next layer unchanged. In technical terms they ride the "residual connection", a bypass that every transformer layer already has, so nothing new needs to be built for the skip.

Here is the part that makes it practical. The model does not let any number of tokens through. It sets a fixed capacity in advance. In the setup the paper found worked best, a routed layer processes only 12.5% of the tokens, which is 256 out of a 2,048-token passage, and the authors applied this to every other layer. The layers in between still process every token.

That fixed cap is the clever engineering choice. Chips like GPUs run fastest when they know the size of the job ahead of time. Because the number of selected tokens is always the same, the computer always does a known amount of work, even though which tokens get picked changes from passage to passage.

The router is not a hand-written rule like "skip punctuation". It is trained along with the rest of the model and learns its own sense of which tokens matter. The paper observes that some tokens engage with almost every layer, while others "route around blocks whenever possible". The researchers also tried picking tokens at random, and that version performed "drastically worse", which tells you the learned choice is doing real work.

Different tokens end up passing through different numbers of layers, so the model is effectively a mix of shallow and deep paths, which is where the name comes from.

What did the paper actually show?

The team trained models ranging from 60 million to 3 billion parameters (parameters are the adjustable numbers a model learns; today's leading chatbots are far larger). They compared MoD models against standard transformers given the same total training budget, testing at three budgets: 6e18, 2e19 and 1e20 FLOPs.

Three results stand out, all from the paper's full text:

• Equal or better quality for the same training spend. The authors report that MoD "improves upon vanilla transformers by as much as 1.5%" on the training objective, a measure of how well the model predicts the next token, when both get the same training FLOPs.
• Less work per step. Because most tokens skip half the routed layers, the models need, in the abstract's words, "a fraction of the FLOPs per forward pass". A forward pass is one trip of the text through the model.
• Faster steps. The paper says the models "can be upwards of 50% faster to step during post-training sampling", meaning when the trained model generates text. It also reports training steps that are upwards of 60% faster on equivalent hardware.

There was a catch the authors had to fix. Picking the "top" tokens means comparing every token in a passage, including ones later in the sentence. When a chatbot writes text one token at a time, those later tokens do not exist yet.

Their solution was a second, tiny predictor that learns to guess the router's decision using only the current token. The paper reports this guess is upwards of 97% accurate and that switching to it caused little loss in quality.

How is mixture of depths different from mixture of experts?

You may have heard of mixture of experts, often shortened to MoE. It is the design behind several large open models, including the one in our DeepSeek-V3 explainer. The two ideas share a family resemblance, and the paper says so directly: MoD borrows MoE's routing logic, but "MoD deploys a single expert which can be dynamically skipped."

The difference is what the router chooses.

Put simply, mixture of experts asks "who should handle this word?" while MoD asks "does this word need handling here at all?"

The two are not rivals. The paper tested a combination it called MoDE and reports that one way of combining them worked "distinctly better" than simply trimming expert capacity in a standard MoE model, though it gives these results as charts rather than headline numbers.

MoD also differs from an older idea called early exit, where a token stops at some layer and never goes deeper. With MoD, the authors note, "a token can skip middle layers" and still be processed by later ones.

What the paper did not prove

It is easy to oversell this research, and earlier coverage (including our own first version of this article) did. Here is what the evidence does and does not support.

It does not show a model running twice as fast. The "upwards of 50%" figure describes how quickly a sampling step can run, and the paper is careful about how it measured it. In its scaling charts, the authors say they "depict normalized FLOPs per forward pass rather than wall-clock step time per se." Fewer calculations usually means faster responses, but real-world speed also depends on memory, software and hardware.

It was tested on small models. The largest model in the study had about 3 billion parameters. That is useful for research but small next to the commercial systems people use daily. Whether the gains hold at much larger sizes is an open question the paper does not answer.

It reports training quality, not real tasks. The 1.5% improvement is on the model's own prediction objective. The paper does not report results on everyday benchmarks such as question answering or coding tests.

Memory savings were not measured. Tokens that skip a layer also skip storing some of that layer's working notes, the cached data explained in our KV cache guide. The authors mention this possible benefit but write, "We did not study this extensively."

There is no public sign it ships in a product. The paper does not say whether Google uses MoD in Gemini or any other service, and we found no public confirmation that a major commercial model uses it.

None of this makes the work less interesting. It makes it what it is: a careful research result showing that a model can learn where to spend its effort, at a modest scale.

What to watch next

The bigger question is whether the idea can be added to existing models cheaply, since few companies want to retrain from scratch. A later paper called Router-Tuning, by researchers including Shwai He and Tao Ge, tries exactly that by training only the routers on a small dataset. Its authors report a 21% speedup with only a 0.2% drop in performance, a smaller gain than the original paper's best case but on models that already existed.

Watch the name, too. In March 2026, a separate team published Mixture-of-Depths Attention, or MoDA, which lets each layer look back at information from earlier layers. Despite the similar name, it solves a different problem, so search results that say "mixture of depths attention" may not be about the 2024 token-skipping method.

For readers who just want the takeaway: AI models have long spent the same effort on every scrap of text. MoD showed, at research scale, that a model can learn to skip the easy parts and lose nothing on its training goal. If that holds at larger sizes, it is one more lever, alongside cheaper hardware and smarter caching, for bringing down the cost of running AI.