Fits in memory, still slow: why bandwidth sets your tok/s

Your model fits in memory. You start it, and the words arrive at a crawl. Fitting is only half the story: how fast the chip can read the model decides the speed, and a dense model reads a lot more than a mixture-of-experts model.

This is the second step after the question of whether a model fits. It explains why a 128 GB AI box writes a 120B-class answer about four times faster than it writes a 27B one.

The rule: speed is bandwidth over bytes

For every token (a token is a word or part of one) the chip must read the active weights from memory once. So the writing speed in tok/s (tokens per second) is roughly:

memory bandwidth ÷ size of the active weights

Bandwidth is how many gigabytes per second the chip can read. Two examples, with round numbers. A 4-bit dense 27B model has a file of about 17 GB, and all of it is read for each word. At 1,792 GB/s that is a ceiling above 100 tok/s; at 273 GB/s it is about 16. Real speeds land below the ceiling, and the selector's estimates account for that.

Dense and mixture-of-experts

A dense model uses all its weights for every token. A mixture-of-experts model (MoE) has many weights but routes each token through only a few of them. According to OpenAI's model card, gpt-oss-120b has about 117 billion parameters in total and about 5.1 billion active per token. The chip reads only the active part for each word, so it behaves like a small model for speed while it needs the memory of a large one.

That flips the usual order. A big-memory, low-bandwidth box is poor at dense models and good at MoE models. A fast card with small memory is the opposite.

What the selector estimates

The chart shows the AI selector's speed estimates for two models at 4-bit and a 32,000-token context, for one user. Two of them are cards with 32 GB or 96 GB, two are 128 GB boxes: the ASUS Ascent GX10 and a Ryzen AI Max+ 395 box.

Look at the dense model first. The RTX 5090 is estimated at about 63 tok/s, an RTX PRO 4500 at about 35, and the two boxes at about 10. A box writes that dense answer at roughly a sixth of the speed of the faster card, which is about the ratio of the bandwidths.

Now the MoE model. The two boxes write gpt-oss-120b at about 42 tok/s (GX10) and 40 tok/s (Ryzen box). That is faster than the same boxes run the 27B dense model by a factor of four, even though the model is much larger. A 32 GB card cannot hold it. It must offload part to system RAM, and the estimate drops to about 20 tok/s on an RTX 5090. One RTX PRO 6000 holds it all and writes about 176 tok/s.

A real-world check

These are estimates, so here is a public data point. A user in the llama.cpp community thread on the DGX Spark ran gpt-oss-120b and reported writing speeds of 46 and 61 tok/s on two software builds, and 34 and 41 tok/s with 32K tokens already in the context. The selector's 42 tok/s for the GX10 sits in that range. These are community-submitted results, not EMD Lab measurements, and they show that a long context slows writing as well.

What offload does

Offload means part of the model sits in ordinary system RAM because the card is too small. For every token, the CPU then works through those layers from system RAM, which is far slower than VRAM, while the GPU waits. That is why a 32 GB card at 20 tok/s on gpt-oss-120b is much slower than a card that holds the whole model, and why the estimate for a dense 70B model on a 32 GB card is a few tok/s. If you are close to the line, a model that fits is better than a faster card that does not hold it.

What this means when you buy

• Unified-memory boxes (GX10, Ryzen AI Max+ 395): choose them for MoE models. gpt-oss-120b and similar large MoE models run at a pace you can work with, on one desk.
• Dense 27B to 31B models: these want a GPU. An RTX PRO 4500 or an RTX 5090 gives several times the speed of a box. Dense 70B models need the 96 GB of an RTX PRO 6000 to run at all at a good pace.
• Check which kind a model is. The model card on Hugging Face says whether it is dense or MoE, and how many parameters are active.

The short version

• Tok/s is roughly bandwidth divided by the active weights read for each word.
• Dense models read everything every time. MoE models read only a few billion parameters, so they are fast on low-bandwidth boxes.
• A 128 GB box writes gpt-oss-120b at an estimated 40 to 42 tok/s but a dense 27B model at only about 10.
• If it does not fit in VRAM, offload to system RAM slows it sharply.

All speeds are selector estimates, except the community thread noted above. Test your own combination in the model finder.