Most spec sheets quote one speed: how fast the model writes. But a local model has two speeds. It reads your prompt, then it writes the answer. For chat the second one matters. For RAG, long documents and agents the first one does, and it is the speed that separates graphics cards from AI boxes the most.
Two speeds: reading and writing
Prompt processing (pp) is the model reading everything you send before it writes the first word. That includes your question, the documents RAG pulled in, and the instructions of an agent. Token generation (tg) is the writing. Both are counted in tok/s (tokens per second; a token is roughly a word or part of one).
The gap between the two is large. Reading is fast because the chip can work on many tokens of the prompt at once. Writing is slow because each new word depends on the last one. In llama.cpp tests, the same chip reads at thousands of tokens per second and writes at tens to hundreds.
Why reading is a compute job
Writing is limited by memory bandwidth: each word needs a full read of the active weights. Reading a prompt is different. The chip reads the weights once and applies them to many tokens at the same time, so the limit becomes arithmetic speed, not memory speed. That is why a graphics card, with a very large number of compute units, leads by a wide margin. An AI box has a capable GPU, but a much smaller one, sharing memory with the CPU.
The wait before the first word is called time to first token. For a short chat message it is a fraction of a second. For a long prompt it is the whole wait.
What the public results show
The chart uses llama.cpp community results for pp512, reading a 512-token prompt, on the same 7B 4-bit model for every chip.
Prompt reading speed, 7B model, 4-bit (llama.cpp pp512)
Tokens per second. Higher is better. One chip each. Ranges are where several results exist.
Source: llama.cpp community results (Llama 2 7B Q4_0, pp512). Not EMD Lab measurements. The RTX PRO 6000 result is the full-power Workstation Edition; the Max-Q card in our AI machines is not in this data.
Two things stand out. The discrete graphics cards read about 14,000 to 16,600 tokens per second, far ahead. And among the unified-memory chips, Apple's M5 Max and M5 Ultra read as fast as or faster than the DGX Spark. So the gap is graphics cards against every unified-memory computer, and the Ryzen AI Max+ 395 is the slowest reader of the group by a good margin.
A worked example: a 32,000-token tender
Suppose you paste a tender document of about 32,000 tokens (roughly 24,000 words) and ask for a summary. Time to first token is simply the prompt length divided by the reading speed. The table is arithmetic on the community numbers above, not a measurement.
| Chip | Reads at | Wait before first word |
|---|---|---|
| RTX PRO 6000 Blackwell | 14,855 to 16,619 tok/s | about 2 seconds |
| GeForce RTX 5090 | 14,073 to 14,970 tok/s | about 2 seconds |
| RTX PRO 4500 Blackwell | 7,252 tok/s | about 4 seconds |
| Apple M5 Max | 3,220 tok/s | about 10 seconds |
| NVIDIA DGX Spark | 3,062 to 3,661 tok/s | 9 to 10 seconds |
| AMD Ryzen AI Max+ 395 | 911 to 1,004 tok/s | 32 to 35 seconds |
Real prompts are worse, for two reasons. The 7B model is small, so a 27B or 120B model reads slower on every chip. And reading slows down as the context grows. The DGX Spark thread shows this for gpt-oss-120b: it reads about 1,740 to 1,960 tok/s on a fresh 2,048-token prompt, and about 1,030 to 1,230 tok/s when 32K tokens are already in the context. Divide 32,000 by those rates and the wait is about 16 to 18 seconds at best and 26 to 31 seconds at the deeper rate. That is still the arithmetic of a community result on one machine. It shows the order of magnitude, not a promise.
Where reading speed decides
- RAG. Retrieval-augmented generation pastes the passages it found into your prompt on every question, so every answer starts with a read. Short chat feels fast; a 20,000-token retrieval does not.
- Long documents. Contracts, tenders, manuals and reports go into the prompt in full.
- Agents and coding assistants. They re-send long instructions, tool results and code on every step. Serving software can cache a prompt it has already read, which helps, but every new tool result and file still has to be read. A slow reader multiplies the wait across the steps.
- Many users. Each person's prompt is read before they get an answer. Reading is compute, and the compute is shared.
A short chat prompt is a different case. A few hundred tokens read in well under a second on any of these chips, and then the writing speed, which follows bandwidth, takes over. If that is your use, a box is fine.
What to pick
For long documents, many users or agents, point to an RTX card or an RTX PRO server. In the data, one RTX PRO 4500 reads about 1.5 to 2 times as fast as the best unified chips, and the RTX PRO 6000 about three times as fast or more. Our AI workstation has two RTX PRO 6000 Max-Q cards, and the Team server has four. For chat-length prompts, one person, and a large mixture-of-experts model on the desk, a unified-memory box such as the ASUS Ascent GX10 is fine; ask us if you want to see how it fits.
The short version
- Reading (pp) is compute-bound and decides the wait before the first word. Writing (tg) is bandwidth-bound.
- Public llama.cpp results put the RTX 5090 and RTX PRO 6000 at about 14,000 to 16,600 tok/s for reading, against about 900 to 4,900 for unified-memory chips.
- A 32,000-token document is a wait of about 2 seconds on an RTX 5090 or RTX PRO 6000 and 10 to 35 seconds on a box, in the 7B arithmetic, and more with bigger models.
- RAG, long documents and agents point to a GPU. Short chat is fine on a box.
All figures are public community results or arithmetic on them. Compare cards and boxes in which hardware runs which AI.