Artificial Analysis 发布 NVIDIA Vera Rubin NVL72 首批 AA-AgentPerf 智能体推理评测

Agentic coding workloads are hard to serve efficiently

Vera Rubin NVL72 leads GB300 NVL72 by up to ~5.7x per MW

Vera Rubin NVL72 keeps the rack-scale design of GB300 NVL72, with 72 GPUs and 36 CPUs in a single NVLink domain. Each Rubin GPU has the same 288 GB of memory as Blackwell Ultra, but 2.4x the memory bandwidth (from HBM4), 1.7x the NVLink bandwidth and 2.3x the dense NVFP4 compute, and the Vera CPU doubles the NVLink-C2C bandwidth to its GPUs, which is the link that carries any KV cache offloaded to CPU memory.

Per GPU	Rubin (Vera Rubin NVL72)	Blackwell Ultra (GB300 NVL72)
GPU memory	288 GB HBM4	288 GB HBM3E
Memory bandwidth	19.2 TB/s	8 TB/s
NVLink bandwidth	3.0 TB/s (NVLink 6)	1.8 TB/s (NVLink 5)
Dense NVFP4 compute	35 PFLOPS	15 PFLOPS
Sparse NVFP4 compute	50 PFLOPS	20 PFLOPS
CPU	Vera, 88 cores	Grace, 72 cores
CPU to GPU link per superchip	1.8 TB/s NVLink-C2C	0.9 TB/s NVLink-C2C
Measured power in these results	0.5 to 1.9 kW	0.3 to 1.0 kW

Specifications from NVIDIA’s Vera Rubin NVL72, Blackwell Ultra and GB300 NVL72 pages as of October 2026. Measured power is each GPU’s own power draw under load (the GPU and its HBM, as reported by NVIDIA’s DCGM/SMI telemetry, excluding CPUs, networking and cooling), across the configs on each system’s frontier for the four models.

Results across the four models

Each chart below plots total throughput per MW of measured GPU power against per-user output speed. Total throughput counts input tokens (cached and newly prefilled) and output tokens, with ~96% of them cached input, and per-user speed is the 25th percentile (p25) across requests excluding time to first token, which is the speed we quote throughout. Each point is a serving config at a set number of agents, and the arrow marks Vera Rubin’s largest lead over GB300.

At peak throughput, Vera Rubin NVL72 serves 272M tokens/s per MW on DeepSeek V4 Pro vs 239M for GB300 NVL72 (1.14x), 571M vs 395M on DeepSeek V4.1 Flash (1.45x), 272M vs 201M on GLM-5.2 (1.35x) and 70M vs 56M on Kimi K3 (1.26x)
Vera Rubin’s lead is largest where GB300’s throughput drops away, reaching ~5.7x on DeepSeek V4 Pro at ~190 tok/s per user, ~5.4x on DeepSeek V4.1 Flash at ~580 tok/s, ~3.0x on GLM-5.2 at ~165 tok/s and ~2.8x on Kimi K3 at ~67 tok/s. Going from ~120 to ~165 tok/s per user on DeepSeek V4 Pro costs GB300 81% of its throughput per MW, compared to 16% for Vera Rubin
Per GPU, Vera Rubin’s lead is wider again: 2.0x to 2.5x at peak throughput, and up to ~12x on DeepSeek V4 Pro at ~165 tok/s per user, ~18x on DeepSeek V4.1 Flash at ~580 tok/s, ~7x on GLM-5.2 between ~200 and ~310 tok/s and ~7x on Kimi K3 at ~176 tok/s
Vera Rubin’s fastest configs set the highest per-user speeds on our leaderboard for DeepSeek V4.1 Flash (906 tok/s, the fastest result for any model), GLM-5.2 (408 tok/s) and Kimi K3 (255 tok/s)

Greater power efficiency lowers the operating cost of serving each token, and as AI datacenters are increasingly limited by the power they can draw rather than the number of accelerators they can install, it also allows more agents to be served out of the same datacenter. At ~165 tok/s per user, for example, the same GPU power budget serves ~5x as many DeepSeek V4 Pro tokens on Vera Rubin NVL72 as on GB300 NVL72. Every point on our leaderboard comes with the serving config behind it in our configurations browser, so teams can start from a tuned setup for their own hardware rather than running their own performance study.

DeepSeek V4 Pro results show the widest lead at high speeds

DeepSeek V4.1 Flash results set the highest throughput and speeds on our leaderboard

GLM-5.2 results show a ~3x lead at faster speeds

Kimi K3 results show a ~2.8x lead at moderate speeds

NVIDIA tuned its Vera Rubin stack for agentic workloads

Overview of the Vera Rubin configs for each model

	DeepSeek V4 Pro	DeepSeek V4.1 Flash	GLM-5.2	Kimi K3
Serving stack	TensorRT-LLM with NVIDIA Dynamo	vLLM with NVIDIA Dynamo	TensorRT-LLM with NVIDIA Dynamo	TensorRT-LLM, and vLLM for 1 agent, both with NVIDIA Dynamo
Weights	NVIDIA NVFP4 checkpoint	DeepSeek release checkpoint (MXFP4)	NVIDIA NVFP4 checkpoint	NVIDIA NVFP4 checkpoint
Prefill and decode	Disaggregated. DEP4 prefill workers, wide expert parallel decode (DEP16 or DEP32), TP4/EP4 decode at 34 agents	Aggregated TP4 or TP2 for 1 and 2 agents. Disaggregated from 64 agents, with one or two TP2 prefill workers and DEP4 decode (or five TP2 decode workers)	Disaggregated. 1 to 3 prefill workers (TP2/EP2 or DEP4), decode on TP8, TP4/EP4 or DEP8	Aggregated TP8 on vLLM for 1 agent. Disaggregated from 16 agents: a TP8 prefill worker with three TP8 decode workers at 16 and 30 agents, and DWDP16 prefill (one or two groups) with DEP16 decode from 512 agents
Speculative decoding	DSpark, 3 to 7 draft tokens	Built-in DSpark drafting, 6 draft tokens	MTP, 4 to 8 draft tokens	DSpark, 3 to 7 draft tokens
KV cache	FP8, offloaded to host memory on prefill workers	Kept in GPU memory, with the model’s Engram memory offloaded to host memory	FP8, offloaded to host memory on prefill workers in all but one config	FP8, offloaded to host memory in the TensorRT-LLM configs
Routing	KV-aware, with session affinity	KV-aware across two prefill workers at 1,024 agents, a single prefill worker otherwise	KV-aware, with session affinity	KV-aware, with session affinity
GPUs per config	32 to 56	2 to 12	10 to 20	8 to 48