Spec sheets for AI PCs lead with one number: 50 TOPS. It sounds like a speed rating for AI. For local chat models it mostly is not. This post explains what the number counts, why it does not predict tok/s, and what to look at instead.
What the 50 means
TOPS means trillions of operations per second, a count of raw arithmetic. On a modern AI PC chip the headline figure usually belongs to the NPU, the neural processing unit, a small engine built to do a few AI jobs on very little power. On the AMD Ryzen AI Max+ 395, AMD rates the XDNA 2 NPU at up to 50 TOPS, and defines TOPS as the peak in an optimal scenario. Spec sheets sometimes also show a bigger "overall" figure that adds the CPU and the graphics part to the NPU. These are real ratings, but a sum of three engines is not a speed you get for a chat model.
Why TOPS does not set your tok/s
When a model writes an answer, it produces one token (roughly a word) at a time. For every token the chip must read the active weights of the model from memory. The chip has plenty of arithmetic to spare. It spends most of its time waiting for memory. So the speed of writing, tok/s, is limited by memory bandwidth, how many GB per second the chip can read.
The NPU, the graphics part and the CPU all share the same memory on these chips. Adding TOPS adds arithmetic, not bandwidth. A chip with twice the TOPS and the same memory writes at about the same speed.
What the public numbers show
The llama.cpp community keeps a standard test, Llama 2 7B at Q4 (a 4-bit model file), and posts tg128, the speed of writing 128 tokens. Place those results next to each machine's memory bandwidth and the pattern is plain. Writing speed follows bandwidth.
Writing speed follows memory bandwidth (Llama 2 7B Q4_0, tg128)
llama.cpp community results, not EMD Lab measurements. Memory bandwidth in GB/s is in each label. The RTX 5090 bar shows the range of its two results.
Source: llama.cpp community results, Llama 2 7B Q4_0, tg128. Bandwidth from NVIDIA, Apple and AMD figures as used in the EMD Lab AI selector hardware data.
It is not a perfect line, because software and chip design differ, but the order matches. The Apple chip has about 2.3 times the bandwidth of the DGX Spark and writes about twice as fast. The RTX 5090 has about 6.5 times the bandwidth and writes about five times as fast. The NPU never enters this chart, because these tests run on the graphics part.
What the NPU is good for
The NPU is not useless. It is built for a different job:
- Windows Copilot+ features and webcam effects such as background blur, which run all day.
- Low-power background jobs and small always-on models, where battery life matters more than speed.
- A small model next to a big one. Run a small helper model on the NPU while the graphics part serves the big model.
AMD's Ryzen AI Software documentation points to llama.cpp for the integrated graphics, not the NPU, so big local chat models run on the graphics part. If you plan to use the NPU for a particular chat model, check AMD's current supported list first.
Where else TOPS misleads
TOPS also hides which kind of arithmetic is counted. Vendors usually quote it for low-precision integer maths, the kind a small NPU does well, while the graphics part works in different number formats. Two TOPS figures on two data sheets are often not counted the same way, so they cannot be compared directly. And even a fair comparison would only describe the arithmetic side, which is the side that is not holding up your chat answer.
There is one more effect. Reading a long document (prompt processing) does lean on arithmetic more than writing does, so compute matters there. That is a job for the graphics part, which is why a discrete GPU reads prompts several times faster than a shared-memory box. Again, the NPU's TOPS figure is not the number that decides it.
What to buy instead of TOPS
Ask two questions. How much memory can the AI use? That decides whether the model fits at all; the selector estimates gpt-oss-120b at about 67 GB with its context. How fast is that memory? That decides the tok/s. A 128 GB box at 256 to 273 GB/s holds gpt-oss-120b and writes it at a usable speed for one person. A card with 32 GB at 1,792 GB/s writes small models much faster, but cannot hold the big one. Neither answer appears in a TOPS number.
So the advice is short: do not buy TOPS, buy memory size and bandwidth. The hardware guide lists both for each platform, and the model finder turns them into an estimated tok/s for the model you pick.
The short version
- "50 TOPS" is the NPU. A larger "overall" figure, where a spec sheet shows one, adds the CPU and graphics part too.
- Chat speed is set by memory bandwidth, and all three parts share one memory.
- Public llama.cpp results line up with bandwidth: about 50 tok/s at 256 GB/s, about 57 at 273, about 120 at 614, about 290 to 300 at 1,792.
- The NPU suits Copilot+ features, background jobs and small always-on models.
- Compare memory size and memory bandwidth, not TOPS.