What a commissioned report contains
We benchmark the models you name across the hardware you name, at more than one context length and up to an agreed concurrency. You receive the raw JSON, a reproducible public report, a signed PDF you can forward, and a permanent URL. Every speed is the median of several independent runs and each individual run is printed beside it. The method is published, negative results are published, you have right of reply and no editorial veto, and commissioned work is excluded from every ranking on this site.
What fits, and what does not
Weights on disk, peak VRAM measured at the card, headroom left at the stated context — and, first-class, every configuration that failed to load. A model list where everything worked has been curated, and a curated list is not a measurement.
| Model | Quant | Weights | Peak VRAM | Headroom |
|---|---|---|---|---|
| gpt-oss 20B | Q4_K_M | 10.8 GB | 11.3 GB | 12.7 GB |
| Qwen3 30B A3B Instruct 2507 | Q4_K_M | 17.3 GB | 18.0 GB | 6.0 GB |
| Qwen3 4B Instruct 2507 | Q4_K_M | 2.3 GB | 3.4 GB | 20.6 GB |
| Qwen3 8B (Q3_K_M) | Q3_K_M | 3.8 GB | 4.7 GB | 19.3 GB |
Speed for one person at a keyboard
Decode and prefill tok/s with the standard deviation across repetitions, plus time to first token. This is the number every benchmark publishes, and on its own it is the wrong number for anyone serving users.
| Model | Decode | σ | Prefill | TTFT p50 |
|---|---|---|---|---|
| gpt-oss 20B | 288.8 tok/s | ± 0.8 | 12,879.8 tok/s | 19.4 ms |
| Qwen3 30B A3B Instruct 2507 | 259 tok/s | ± 1.2 | 9,923.8 tok/s | 15.5 ms |
| Qwen3 4B Instruct 2507 | 258.4 tok/s | ± 0.4 | 17,308.2 tok/s | 7.1 ms |
| Qwen3 8B (Q3_K_M) | 186.5 tok/s | ± 0.2 | 11,108.7 tok/s | 9.3 ms |
Speed when it is serving a product
Aggregate throughput measured over the seconds the server was actually saturated, per-stream rate, and the latency tail — with the number of requests behind each percentile stated, because a p95 of a handful of requests is the largest of that handful. Cost per token computed from a single stream can be an order of magnitude too high.
| Streams | Aggregate | Per stream | TTFT p50 | Samples |
|---|---|---|---|---|
| 1 | 264.6 tok/s | 264.6 tok/s | 31.7 ms | 20 |
| 2 | 211.7 tok/s | 105.8 tok/s | 57.6 ms | 20 |
| 4 | 322 tok/s | 80.5 tok/s | 77.9 ms | 20 |
| 8 | 381 tok/s | 47.6 tok/s | 116.8 ms | 24 |
| 16 | 862.1 tok/s | 53.9 tok/s | 132.6 ms | 48 |
Whether it still works at that speed
A deterministic probe set run against every quantisation, beside the speed that describes it. A quant that has damaged the model is faster than one that has not, so speed alone rewards damage. It is a smoke test, not a quality ranking, and the report says so wherever the number appears.
| Model | Quant | Score | Looping |
|---|---|---|---|
| gpt-oss 20B | Q4_K_M | 38% | yes |
| Llama 3.1 8B Instruct | Q4_K_M | 50% | no |
| Qwen3 8B (Q5_K_M) | Q5_K_M | 50% | no |
| Qwen3 8B (Q8_0) | Q8_0 | 50% | no |
What context length costs
The number that decides whether a model you can load is a model you can use. Measured peak against the KV formula every calculator applies, so the gap between them is visible instead of assumed.
| Context | Weights | Theoretical KV | Measured peak | Decode at depth |
|---|---|---|---|---|
| 4,096 | 10.8 GB | 0.2 GB | 11.1 GB | 271.9 tok/s |
| 16,384 | 10.8 GB | 0.8 GB | 11.4 GB | 245.6 tok/s |
What happens on a card too small for it
Most buyers' customers do not have the card in the report. The offload curve is what they will actually experience, measured rather than modelled — with the CPU, the thread count and the RAM stated, because below the fully-resident row this is a measurement of the host as much as the card.
| On GPU | Layers | Decode | vs fully resident |
|---|---|---|---|
| 100% | 999 / 24 | 285.6 tok/s | 100% |
| 75% | 18 / 24 | 90.1 tok/s | 32% |
| 50% | 12 / 24 | 55.9 tok/s | 20% |
| 25% | 6 / 24 | 41.1 tok/s | 14% |
The configuration your readers are actually running
LM Studio and Ollama enable flash attention by default; a raw llama.cpp invocation does not. Both populations read benchmarks, and they are not running the same thing — so it is measured both ways, at depth, where the difference shows up.
| Flash attention | Decode | Peak VRAM |
|---|---|---|
| off | 250.7 tok/s | 11.8 GB |
| on | 272.1 tok/s | 11.4 GB |
Which quantisation is worth shipping
Same weights, same rig, different bits — VRAM, speed and the accuracy probes at every rung. The trade is usually described in quality terms and priced in VRAM; the speed column is the one people forget, and nobody publishes it next to a correctness check.
| Quant | Weights | Peak VRAM | Decode | Accuracy |
|---|---|---|---|---|
| Q3_K_M | 3.8 GB | 4.7 GB | 186.5 tok/s | 63% |
| Q4_K_M | 4.7 GB | 5.5 GB | 163.3 tok/s | 69% |
| Q5_K_M | 5.4 GB | 6.1 GB | 144.7 tok/s | 50% |
| Q6_K | 6.3 GB | 6.9 GB | 128.3 tok/s | 56% |
| Q8_0 | 8.1 GB | 8.6 GB | 103.7 tok/s | 50% |
What it costs to own, not to rent
Board power sampled at the card during generation rather than taken from a spec sheet, as watts, tokens per watt and kWh per million tokens. A rented pod's hourly rate stops mattering the day the buyer owns the machine; this column does not.
| Model | Avg power | Peak temp | Tokens/watt | kWh per Mtok |
|---|---|---|---|---|
| Qwen3 30B A3B Instruct 2507 | 181.4 W | 45°C | 1.43 | 0.195 |
| gpt-oss 20B | 209.7 W | 47°C | 1.38 | 0.202 |
| GLM 4.7 Flash | 186.6 W | 44°C | 1 | 0.278 |
| Qwen3 4B Instruct 2507 | 267.7 W | 49°C | 0.97 | 0.288 |
Cost per million tokens, single-stream and batched
Computed at four utilisation assumptions, because no deployment generates tokens every second it pays for. The gap between the single-stream and batched columns is the whole reason a load test exists.
| Model | Single stream | Batched (peak) | Batching worth |
|---|---|---|---|
| Qwen3 4B Instruct 2507 | $0.74 | $0.15 | 5.1× |
| Llama 3.1 8B Instruct | $1.12 | $0.16 | 7× |
| Qwen3 8B (Q3_K_M) | $1.03 | $0.17 | 6.1× |
| Qwen3 8B | $1.17 | $0.18 | 6.6× |
Break-even against an API price you name
The volume above which running it yourself wins, and the sustained utilisation that implies — including the case where the machine cannot physically produce that volume in a month, which is printed as “never” rather than omitted.
Everything that failed, and the method to reproduce it
Each failure with the stage and the error, then the pinned engine build, driver, host CPU and thread count, every axis swept and the raw JSON beside the prose. A methodology bug is an issue, not a secret.
No excerpt: every model in this particular run loaded, so its failure table is empty. That is a property of the list, not of the format — a run whose ladder stops somewhere is the more useful one, and the row that stops it is usually the most quoted line in the report.
- NVIDIA only. The harness builds CUDA. AMD, Intel Arc and Apple Silicon cannot be measured at all today — not slowly, not approximately.
- llama.cpp and vLLM. Figures for other serving stacks are a conservative floor and are labelled as one.
- No operating costs. Ops labour, storage and egress are deliberately excluded from the cost model and stated as excluded, because guessing at someone else’s payroll is not a measurement.
- Not a quality ranking. The accuracy probes catch a broken configuration. They cannot tell you which of two healthy models is smarter.
Negative results are published. You have right of reply, printed in full, and no editorial veto. The raw data is released with the report. Ranking on this site is never for sale, and commissioned runs are excluded from it. These four clauses are the product: a report a vendor could have edited is worth nothing to the engineer it is aimed at, and that is the entire reason it is worth paying for.
What it costs and how to commission one →
Or read a full published report (NVIDIA GeForce RTX 4090) → · raw JSON · llama.cpp b10156