A 128 GB AI box can run a 120B model on a desk, and for one developer it feels great. Then a second, third and sixth colleague start asking it things. At some point the box stops being the right tool.
This post is about that point. The short answer is that memory room and speed are two different limits, and a box with lots of the first can run short of the second.
Why more users cost more
Serving a model to several people at once adds two costs on top of the shared model file.
- Each chat needs its own KV cache. That is the model's working memory for one conversation. The weights are loaded once and shared, but every open conversation keeps its own cache, and it grows with the length of the text. We explained the arithmetic in our guide to GPU memory for local LLMs; here we only use the results.
- Everyone shares the memory bus. To write each word, the chip reads the active part of the model from memory. Serving software such as llama.cpp (parallel slots) or vLLM (batching) lets several chats take turns on the same hardware. The memory bus is the road they all drive on, so more cars means slower trips for each.
What the box and the tower offer
Two kinds of machines sit at the two ends. A 128 GB box, such as the ASUS Ascent GX10 (NVIDIA's GB10 chip) or a Ryzen AI Max+ 395 mini PC, has unified memory: the CPU and GPU share one pool. NVIDIA lists 273 GB/s for the DGX Spark family, the same chip as the GX10, and it runs full CUDA and vLLM. The Ryzen AI Max+ box has similar memory speed, no CUDA, and vLLM is still experimental on it. On Windows the GPU may use up to 96 GB of the pool, by the selector's figure; on Linux more, after the kernel setting described in AMD's guide.
A tower with an RTX PRO 6000 Blackwell Max-Q has 96 GB of GDDR7 at 1,792 GB/s, per NVIDIA. That is about 6.5 times the memory speed of the box. Two cards give 192 GB in one tower.
The numbers: room is fine, speed is not
The selector calculates both limits for two popular models at a 32,000-token context, which is roughly 24,000 words in a chat. "Chats that fit" is how many such conversations fit in memory. The tok/s figure (tokens per second, roughly words per second) is for one user.
| Hardware | gpt-oss-120b speed | Chats that fit | Qwen3.8 27B speed | Chats that fit |
|---|---|---|---|---|
| 128 GB box, GB10 (GX10) | about 42 tok/s | 40 | about 10 tok/s | 41 |
| 128 GB box, Ryzen AI Max+ 395 | about 40 tok/s | 44 | about 10 tok/s | 43 |
| 1 × RTX PRO 6000 Max-Q | about 176 tok/s | 24 | about 63 tok/s | 33 |
| 2 × RTX PRO 6000 Max-Q | about 176 tok/s | 102 | about 63 tok/s | 74 |
Read the table twice. The boxes can hold more chats than one RTX PRO 6000, because they have 115 to 120 GB usable against 95 GB. But gpt-oss-120b writes about four times slower on the box, and the dense Qwen3.8 27B about six times slower. A dense model uses all its weights for every word, so it leans hardest on the memory bus.
gpt-oss-120b, one user, 32K context: estimated writing speed
Estimate. Tokens per second. Higher is faster. This is the speed with nobody else on the machine.
Source: EMD Lab AI selector estimate, calculated from model files and memory bandwidth. Not a measurement.
Now remember that the "chats that fit" column is a memory limit only. When ten people write at once, the box serves all of them from the same memory bus. Batching raises the total output, but each person gets less than the one-user speed, and the text arrives more slowly for everyone. A GPU server starts from a number about four times higher, so the same crowd feels much better.
Reading the prompt delays everyone
There is a second cost that memory counting hides. Before the first word, the model must read the whole prompt: your question plus any document or chat history. That is prompt processing, written pp in llama.cpp results, and it is limited by compute, not only by memory. Writing is tg, token generation.
Public llama.cpp results on the DGX Spark show the effect for gpt-oss-120b. Reading slows when the context is already long:
| Build | Reading, empty | Reading, 32K deep | Writing, empty | Writing, 32K deep |
|---|---|---|---|---|
| 22 Oct 2025 | 1,737 | 1,232 | 45.9 | 34.3 |
| 2 Nov 2025 (after a kernel update) | 1,956 | 1,027 | 60.6 | 40.6 |
Reading drops by roughly 30 to 45% at 32K depth, and writing drops too. At about 1,000 tok/s, a 32K-token prompt takes on the order of half a minute to read, as rough arithmetic. If several people send long prompts together, the box does that reading in a queue, so everybody sees a longer wait for the first word. A discrete GPU reads far faster, so the queue is short. This is why a box that suits a developer working on one big file can feel slow in a shared office.
A rule of thumb
- One developer, or one to five people who ask in turns, with a mixture-of-experts model such as gpt-oss-120b: a 128 GB box is a good fit. Choose the GX10 if you need CUDA and vLLM, for example for fine-tuning. The Ryzen box is the leaner option, without CUDA. Both are "Ask us" on our site: tell us the job and we will advise.
- More than five people at once, long documents, or dense models: move to a GPU server. The QNAP AI server pairs a 32 GB or 96 GB card with twelve SSD bays for the files themselves. The AI workstation has two RTX PRO 6000 Max-Q cards, 192 GB in total.
The counts in the table are estimates for memory only. How many people feel "fast" also depends on the serving software and on how often your team types at the same moment. That is why we size it together with you.
The short version
- Every chat adds its own KV cache to the shared model. Memory room sets how many chats can be open.
- Speed per person falls as people share one memory bus. A 128 GB box starts from about a quarter of the writing speed of a 96 GB RTX PRO 6000 on gpt-oss-120b (estimate).
- Long prompts make everyone wait for the first word, and boxes read prompts slowly compared with discrete GPUs.
- One to five people at once: a box is a fine fit. More than five: use a GPU server.