Home / Solutions / Which hardware runs which AI?
Which hardware runs which AI?
Pick a graphics card, a multi-GPU build or an AI box and see which open models fit, at what compression, roughly how fast, and what each one is good for. Or pick a model and see every machine that runs it.
Data as of 23 Oct 2026. All speeds are single-user estimates, LLMs are sized at 32K context, and none of the figures is an EMD Lab measurement. Looking for the right AI for your work instead? Try Which AI are you?
The short version
llama.cpp is the program that actually runs the model. CUDA, ROCm, Vulkan, SYCL and Metal are the different ways it talks to the graphics chip. You pick the backend that matches the card; the model file (GGUF) stays the same.
Ollama, LM Studio and Jan are friendly apps built on top of llama.cpp. vLLM and SGLang are a different kind of engine, built to serve many users at once. Image, video, OCR and segmentation tools (ComfyUI and friends) don't use llama.cpp at all; they run on PyTorch, which is where NVIDIA's lead is biggest.
The pieces
llama.cpp
Open-source C/C++ engine for GGUF model files, usually compressed to 2–8 bits per weight (the Q4, Q8 labels). Runs on nearly anything. Best for one user or a small team, mixed hardware, and squeezing big models into limited memory. Its team joined Hugging Face in February 2026.
CUDA
NVIDIA's own platform. The fastest and best-supported path: every AI tool (llama.cpp, vLLM, PyTorch, ComfyUI, TensorRT) works here first. Strongest prompt processing, which matters for long documents, RAG and coding agents.
ROCm (HIP)
AMD's answer to CUDA. Needed for PyTorch, ComfyUI and vLLM on AMD. Works only on the GPU, OS and version combinations AMD lists: R9700, RX 9070/9060, RX 7900 series, Instinct MI100 and newer. Linux first. Usually wins on long prompts.
Vulkan
A cross-vendor graphics API that llama.cpp also uses for compute. Works on NVIDIA, AMD and Intel, including older cards that ROCm no longer supports. On AMD it often matches or beats ROCm on generation (one R9700 test: 153 vs 98 tok/s on a 7B model). Only helps llama.cpp-based apps; it does not run PyTorch or vLLM. Ollama turns it on by default when installed.
SYCL (oneAPI)
Intel's path for Arc cards in llama.cpp. On the B70, SYCL tends to win generation and Vulkan wins prompt processing, so benchmark both. PyTorch XPU handles image and video; vLLM runs through Intel's beta "LLM Scaler" fork.
Metal / MLX
Apple Silicon's GPU path. MLX is Apple's own framework and is often the fastest way to run models on a Mac Studio.
vLLM / SGLang
Built for many simultaneous users (batching). CUDA first, ROCm supported, Intel beta. The whole model must fit in GPU memory. Tensor parallel needs the GPU count to divide the model's attention heads, so use 2 or 4 cards, not 3.
Axelera Voyager SDK
Runs Axelera's vision model zoo (YOLO-class detection, tracking, segmentation). No llama.cpp, Ollama, vLLM or PyTorch support.
What runs where
| Hardware | llama.cpp (chat LLMs) | vLLM (many users) | PyTorch (image, video, OCR) | Fine-tuning |
|---|---|---|---|---|
| GeForce RTX 50 · RTX PRO Blackwell | CUDA · best | Full | Full | Full |
| DGX Spark · ASUS Ascent GX10 | CUDA (Arm Linux) | Strong at concurrency | Works, compute-limited | Main reason to buy |
| RTX Spark (Windows on Arm) | Claimed by NVIDIA | Unknown | Unknown | Unknown |
| Radeon AI PRO R9700 · RX 9070 · RX 7900 · W7900 | ROCm + Vulkan | Works; rough edges on RDNA 4 | Works on Linux, slower | Limited |
| Ryzen AI Max+ 395 / Ryzen AI Halo | Vulkan or ROCm | Weak (314 vs Spark's 2,760 tok/s) | Slow | Limited |
| Intel Arc Pro B70 · B60 · B580 | SYCL + Vulkan | Beta (LLM Scaler) | PyTorch XPU | Limited |
| Mac Studio M5 Max / Ultra | Metal / MLX | No (MLX servers instead) | Works, slow for video | MLX only |
| Axelera Metis | No | No | No (Voyager SDK) | No |
Which backend, for which card, with which models
| Card family | Use this backend | Works best with | Avoid |
|---|---|---|---|
| NVIDIA GeForce / RTX PRO | CUDA for everything; vLLM for teams | Any model that fits. Dense 27–31B, coding agents, long-document RAG, all image and video tools | Nothing in particular; capacity is the limit |
| AMD R9700 / RX 9070 / RX 7900 | llama.cpp: Vulkan first for chat, ROCm for long prompts and anything PyTorch | GGUF LLMs, MoE models, OCR, Whisper, image generation | Heavy video generation; assuming CUDA-only ComfyUI nodes will work |
| Ryzen AI Max / Ryzen AI Halo | llama.cpp (Vulkan or ROCm), Lemonade/LM Studio | MoE with few active parameters: gpt-oss-120b class, Gemma 4 26B-A4B, Qwen3-Coder; Qwen3.8-Flash-Next only below Q4 | Dense 27B+ for chat, long documents, many users, video |
| Intel Arc B70 / B60 / B580 | llama.cpp SYCL for generation, Vulkan for prompt speed; OpenVINO for vision and speech | Mid-size MoE and ≤32B dense at Q4–Q6, Whisper, embeddings | Multi-user serving until vLLM support leaves beta |
| DGX Spark / GX10 | CUDA, vLLM, NVIDIA's playbooks | MoE up to ~120B at Q4, fine-tuning, multi-user serving, long prompts | Dense 27B+ models for chat (bandwidth-bound) |
| Mac Studio | MLX or llama.cpp Metal | Single-user large MoE; the 512 GB Ultra for DeepSeek and GLM at ~2-3 bit | Video generation, CUDA-only tools |
| Axelera Metis | Voyager SDK | Camera analytics: detection, tracking, counting | Every model on this page |
Five rules for multi-GPU AI builds
More cards add memory; only tensor parallel adds speed
llama.cpp's default layer split gives each card a block of layers that work one after another, so one user's speed stays close to one card's. vLLM tensor parallel on 2 or 4 identical cards does add speed: this guide (like the Which AI are you? page) assumes about 1.45× for 2 cards and 2.2× for 4. Row split moves more data over PCIe and is usually slower without NVLink.
Tensor parallel wants 2 or 4 cards
vLLM's tensor parallel needs the attention-head count to divide by the number of GPUs, and 3 rarely does. Over PCIe keep it at 2; four PCIe cards give about 1.5–2× the throughput, not 4×. For 3-card builds use llama.cpp layer split or pipeline parallel.
Big MoE models can live in system RAM
MoE models only use a few experts per token, so llama.cpp can keep idle experts in RAM (--n-cpu-moe, plus the expert cache --moe-cache-mib added 7 Oct 2026). One RTX 5090 ran Qwen3.8-Flash-Next at 31–68 tok/s this way with server RAM. Prompts slow down about 5×, so this suits chat, not long documents.
RAM speed matters for offload
Dual-channel desktop DDR5 (~80 GB/s) is the bottleneck when experts sit in RAM. 8-channel workstation or server memory is several times faster. This guide uses the selector's platforms: AM5 desktop (75 GB/s, up to 256 GB), Threadripper TRX50 (180 GB/s, 256 GB) for 2× RTX 5090 and single RTX PRO 5000/6000, and Threadripper PRO WRX90 (350 GB/s, 512 GB) for 3–4 cards and 2× RTX PRO 6000.
Image and video don't pool
Diffusion, OCR, segmentation and speech models load onto one card. Extra cards let you run more jobs at once; they don't let one job use a bigger model. Size these by the memory of a single card.
Power, slots and platform
Rules of thumb for quoting. GPU power is board power at stock settings; leave 30–40% PSU headroom.
| Build | Pool | GPU power | PSU | Platform notes |
|---|---|---|---|---|
| 2× RTX 5090 | 64 GB | 1,150 W | 1,600 W+ (or cap each at ~400 W) | x8/x8 desktop board works; triple-slot coolers need wide spacing and strong airflow |
| 3× RTX 5080 / 5070 Ti / 5060 Ti 16GB | 48 GB | 540–1,080 W | 1,200–1,600 W | Threadripper (TRX50/WRX90) or Xeon W for lanes; risers or open frame for consumer coolers |
| 4× RTX PRO 4000 | 96 GB | 560 W | 1,000–1,200 W | Compact 140 W cards; the easiest 4-card build |
| 4× RTX PRO 6000 Max-Q | 384 GB | 1,200 W | 1,600–2,000 W | WRX90 or Xeon W, blower-style cards; Workstation Edition would be 2,400 W |
| 4× Radeon AI PRO R9700 | 128 GB | 1,200 W | 1,600 W+ | WRX90 or Xeon W; Linux for ROCm multi-GPU |
| 4× Arc Pro B60 / B70 | 96 / 128 GB | 480–920 W | 1,200–1,600 W | Workstation board; dual-B60 boards need x8/x8 bifurcation |
The platform parts for these builds (ASUS and ASRock workstation boards, Seasonic and FSP 1,600 W+ supplies, Fractal and Phanteks cases, Supermicro for 4–8 GPU servers) all come from brands we distribute.
Axelera cards: can they stack?
Yes for capacity of work, no for model size. You can fit several Metis cards in one machine to process more camera streams or run more vision models side by side. Their memory does not pool into one large model the way llama.cpp pools GPU memory, and none of the LLMs, image or speech models in this guide run on them.
Axelera's own way to scale up is a 4-chip card (856 INT8 TOPS, 16 or 64 GB) and multi-chip servers. The current 1-chip store listing shows 16 GB; we could not confirm a current 8 GB version. Pair them with a GPU box that handles the language work.
How the numbers are worked out
- Same engine as the Which AI are you? page. LLM memory and speed use the same formulas and the same model files (exact GGUF file sizes and KV-cache constants from each model's config). Models that are not in those files (Kimi K3, MiMo-V2.6-Pro, LFM2.5, Qwen3 4B, Qwen3-VL-8B, MedGemma) use sizes estimated from their parameter counts and are marked "estimated size".
- Memory a model needs = model file + KV cache for one conversation at 32K tokens + 1.2 GB + 3% of the weights runtime overhead, plus 0.6 GB per extra GPU.
- Memory available = (card memory − 0.6 GB) × cards. Unified-memory boxes use the GPU-usable figure: 115 GB for DGX Spark/GX10 and 120 GB for Strix Halo (Linux; 96 GB on Windows), as in the selector; the others as reported.
- Recommended compression = the best quality from Q8_0 down that still reaches 20 tok/s ("Fast"). BF16 is shown as the ceiling, never as the recommendation. Q3_K_M and the ~2-bit "smallest usable" quants lose noticeable quality; when one of those is the pick, the row also shows the Q4_K_M-with-offload alternative.
- Speed = bytes read per token ÷ (efficiency × bandwidth × multi-GPU scaling) + a fixed per-token time. Efficiency is calibrated in the selector on 25 measured results: NVIDIA 0.85 / 3.1 ms, AMD 0.80 / 1.5 ms, Intel 0.60 / 0.5 ms, unified boxes 0.78 / 5.4 ms (mostly within ±12%; small MoE ±25%). Macs use the unified-box values. Multi-GPU: 2 or 4 identical NVIDIA/AMD cards assume vLLM tensor parallel (1.45× / 2.2×), 3 cards and Intel assume llama.cpp layer split (0.92×). Speculative decoding (MTP) can roughly double some models.
- Offload (discrete GPUs only): the part that doesn't fit runs from system RAM at 50% of the platform's RAM bandwidth plus 5 ms per token. Prompt processing is much slower than the generation figure suggests. llama.cpp's new expert cache can beat these estimates on MoE models.
- Image, video, speech, OCR and embedding needs are this guide's own estimates per card, marked "estimated size".
- "Quadro": NVIDIA retired that name. The 2000/4000/5000/6000 cards here are the current RTX PRO Blackwell generation. Older Quadro RTX 4000/5000/6000 (2018, 8–24 GB) are a much slower generation.
- Estimates, not tests: every speed on this page is an estimate for one user, worked out from the model files and the published memory bandwidth of each card. None of these figures is an EMD Lab measurement.
- Prices are not shown on this page. Use the button on a card to ask us about a machine.
- Not estimated: RTX Spark (bandwidth unpublished) and DGX Station (bandwidth figures unpublished).
Sources
AI benchmarks: what more GPU memory and bigger models get you