1x L40S 48GB
Before you spend $7,500 on this card, find out if it does what you need.
21 models loaded and timed on this machine, in one session, on one llama.cpp build. Read a whole one first if you like: the RTX 4090 and RTX 3090 reports are open in full.
One payment for this report, not a subscription. It stays yours, and it does not expire. You will be asked to sign in first, so the report stays attached to your account.
- Does Llama 3.3 70B Instruct run on it?
- It did, at Q4_K_M — 41.0 GB of 45 GB at peak
- And how fast, model by model?
- Decode, prefill and TTFT p50/p95 on 21 models
- Which quantisation is worth running?
- 2 models run across several rungs; Q4_K_M, Q3_K_M, Q5_K_M, Q6_K, Q8_0 measured on this card
- How long a prompt before it slows down, or stops fitting?
- Decode and peak VRAM at 4K → 32K, on 15 models
- And when a model is bigger than the card?
- tok/s at each resident fraction, 15 models spilled into system RAM
- How many people can use it at once?
- 1 → 16 streams: aggregate tok/s, per-stream tok/s and TTFT p95
- Does the quantisation cost answers?
- Probed at temperature 0 on 21 models, scored per category
- What does it cost to keep running?
- Watts at the card, and dollars per million tokens, single stream and batched
Decode only, single stream, for the fastest 5. The remaining 16 models, and every other measurement, are listed below.
What is in this report
- · Llama 3.1 8B Instruct · Q4_K_M
- · Qwen3 4B Instruct 2507 · Q4_K_M
- · Qwen3 8B · Q4_K_M
- · Qwen3.5 9B · Q4_K_M
- · Gemma 3 12B Instruct · Q4_K_M
- · Qwen3 14B · Q4_K_M
- · gpt-oss 20B · Q4_K_M
- · Mistral Small 3.2 24B Instruct · Q4_K_M
- · Gemma 3 27B Instruct · Q4_K_M
- · Qwen3 30B A3B Instruct 2507 · Q4_K_M
- · GLM 4.7 Flash · Q4_K_M
- · Qwen3 8B (Q3_K_M)
- · Qwen3 8B (Q5_K_M)
- · Qwen3 8B (Q6_K)
- · Qwen3 8B (Q8_0)
- · Gemma 3 12B Instruct (Q5_K_M)
- · Gemma 3 12B Instruct (Q8_0)
- · Mistral 7B Instruct v0.3 · Q4_K_M
- · Phi-3 Medium 14B Instruct · Q4_K_M
- · Qwen2.5 32B Instruct · Q4_K_M
- · Llama 3.3 70B Instruct · Q4_K_M
- × Qwen2.5 72B Instruct · Q4_K_M
Rows marked × failed to load or to run on this card. They are in the report with the stage they failed at, because a model that does not run is a result.
EVERY FIELD, MEASUREMENT BY MEASUREMENT
- · Decode, single stream, tok/s
- · Prefill, tok/s
- · TTFT p50 and p95, ms
- · Peak VRAM reported by the driver
- · Weight size on disk, and load time
- · Rungs measured: Q4_K_M, Q3_K_M, Q5_K_M, Q6_K, Q8_0
- · 2 models run across several rungs on this card
- · Decode measured with the cache filled to 4K, 8K, 16K, 32K
- · 15 models swept, with peak VRAM at each depth
- · 15 models run at several resident fractions
- · tok/s at each fraction, against the fully resident rate
- · Streams: 1, 2, 4, 8, 16
- · Aggregate tok/s, per-stream tok/s and TTFT p95 at each level
- · 21 models load-tested
- · 21 models probed at temperature 0
- · Score per category, and a separate looping check
- · Watts sampled at the card for 21 models
- · Cost per million tokens, single stream and batched
- · The card's decode constants: fixed ms per token, and implied bandwidth
- · Fixed VRAM overhead before the weights — the intercept a fit calculator assumes is zero
- · TTFT p95 against concurrent streams
- · Each with its n and its correlation; a weak fit is quoted without a line
- · GPU: NVIDIA L40S
- · llama.cpp build b10156
- · Driver 570.124.06
- · Run finished 2026-08-30
- · 2.2 hours on the machine
- · This page, in full, at this URL
- · A PDF with the same tables and charts
- · The underlying JSON, through the API, once the report is yours
- · No expiry, and no subscription to keep reading it
Measured on a rented machine, not estimated and not supplied by a vendor. How the runs are taken, and what each figure does and does not cover, is on the methodology page.