You want to run a language model on your own machine, so your documents stay in the building. Before you look at speed or price, ask one question: will the model fit in the graphics card's memory?
If it fits, answers come fast. If it does not, part of the model runs from ordinary system memory and speed drops hard. So memory is the first number to get right.
What takes up GPU memory
Three things fill the card:
- The model weights. This is the model file. A 27B model in 4-bit form is about 17 GB. The same model in 8-bit form is about 29 GB.
- The context, or KV cache. This is the model's working memory for the conversation. It grows with the length of the text and with every person using the model at the same time.
- Runtime overhead. The software needs buffers of its own. We count about 1.2 GB plus 3% of the weights.
Add the three together and you have the memory the model needs. Keep a little free on top, because the card also drives your display.
Smaller files: what 4-bit means
Most people run models in a compressed form called quantization. A 4-bit file (often labelled Q4_K_M) is about a third of the size of the original 16-bit model, and for chat and writing the quality loss is small. An 8-bit file (Q8_0) is closer to the original but needs about 70% more memory than 4-bit. For a first machine, plan around 4-bit.
How much memory popular models need
The chart shows the total for six open models at 4-bit, with a 32,000-token context (roughly 24,000 words) and one user. The two lines mark the usable memory of a GeForce RTX 5090 and of one RTX PRO 6000 Blackwell Max-Q.
GPU memory needed, 4-bit model, 32K context, one user
Estimate. Lower is easier to fit.
- RTX 5090 usable memory (31.4 GB)
- RTX PRO 6000 Max-Q usable memory (95.4 GB)
Source: EMD Lab AI selector estimate, calculated from model file sizes and configuration files. Not a measurement.
Usable memory is the card's memory minus 0.6 GB that the driver keeps for itself.
Context length changes the answer
Long documents need a long context. Look at what happens as you go from 8,000 to 128,000 tokens:
| Model | 4-bit file | 8K context | 32K context | 128K context |
|---|---|---|---|---|
| Qwen3.5 9B | 6.2 | 7.9 | 8.7 | 11.9 |
| Gemma 4 12B | 7.7 | 9.6 | 10.0 | 11.6 |
| gpt-oss-20b | 11.7 | 13.4 | 14.0 | 16.4 |
| Qwen3.8 27B | 17.4 | 19.9 | 21.5 | 27.9 |
| Llama 3.3 70B | 42.5 | 47.7 | 55.7 | 87.9 |
| gpt-oss-120b | 62.8 | 66.2 | 67.1 | 70.7 |
Models differ a lot here. Llama 3.3 70B grows by 40 GB between 8K and 128K, because its design stores a lot per token. gpt-oss-120b grows by only about 4.5 GB. So do not judge a model by its file size alone.
More users, more memory
Every person chatting at the same time needs their own context. The weights are shared. For Qwen3.8 27B at 4-bit and a 32K context, the estimate is 21.5 GB for one user, 28.4 GB for four and 37.6 GB for eight. For gpt-oss-120b it is 75.6 GB for eight users. If a team will use the model together, count the users before you choose the card.
Memory first, then speed
Once the model fits, speed depends mostly on how fast the card can read its own memory. Public llama.cpp results show how current cards compare on a small model, a 7B model in 4-bit form:
Text generation speed, 7B model, 4-bit
Tokens per second. Higher is better. One card.
Source: llama.cpp community results (Llama 2 7B Q4_0, token generation). Not EMD Lab measurements. Results are for the full-power RTX PRO 6000; the Max-Q version used in our AI machines is not in this data.
Bigger models read more data for every word, so they are slower. The selector estimates the speed for the machines below, and it is usually within about 12% of published results, with small mixture-of-experts models less certain.
Which EMD Lab machine fits
| Model | Studio, RTX 5090 (32 GB) | AI, 2 × RTX PRO 6000 Max-Q (192 GB) |
|---|---|---|
| Qwen3.5 9B | Fits, about 125 tokens/s | Fits, about 125 tokens/s |
| Gemma 4 12B | Fits, about 115 tokens/s | Fits, about 115 tokens/s |
| gpt-oss-20b | Fits, about 200 tokens/s | Fits, about 200 tokens/s |
| Qwen3.8 27B | Fits, about 65 tokens/s | Fits, about 65 tokens/s |
| Llama 3.3 70B | Does not fit. About 2 tokens/s from system memory | Fits, about 25 tokens/s |
| gpt-oss-120b | Partly in system memory, about 20 tokens/s | Fits, about 175 tokens/s |
Studio with an RTX 5090: models up to about 30B
The 32 GB card runs everything up to Qwen3.8 27B with room for a long context or a few users. It is the right start for one person or a small group trying models on their own documents. The standard Studio build uses an RTX PRO 4500 Blackwell, which also has 32 GB; you can switch the card to the RTX 5090 in the configurator.
AI: 70B and 120B models
Two RTX PRO 6000 Max-Q cards give 192 GB in one tower. Llama 3.3 70B fits even in 8-bit form at 32K context, an estimated 89 GB. gpt-oss-120b fits with room for many users. This is the machine for a department that wants a large model and long documents.
Team: a shared server
The Team build has four RTX PRO 6000 Max-Q cards, 384 GB in total. Use it when many people share one model. Our configurator estimates about 1,880 W at the wall for this build, so it needs its own circuit; check your power supply and breaker before you order. How many people can chat at once also depends on the serving software, so we size this one together with you.
The short version
- Add the model file, the context and the overhead. That is the memory you need.
- Multiply the context part by the number of people using the model together.
- Buy memory before speed. A model that fits in a slower card beats a faster card that cannot hold it.
- Plan around 4-bit models and about 30 GB for one person. Go to 96 GB or more for 70B and larger models.
The numbers above are estimates. They show the size of the job, not a promise. Try your own model and context in the model finder.