FitMyLLM
← all rig reports
▸ TAKE THE REPORT WITH YOU

Everything below is also a PDF — every table, every chart, the machine and the method, in one document you can keep or forward. The raw JSON is the same measurements as data, for anyone who would rather check them than read them.

↓ RAW JSON
▸ MEASURED

NVIDIA GeForce RTX 3090

MODELS MEASURED
17
DATA POINTS
551
HOURS ON THE BENCH
1.8
MEASURED ON
2026-07-31
PROVENANCE
24.0 GB VRAMdriver 580.126.20llama.cpp b10156provider runpodcontext 4,096

Every figure below came off this machine. Nothing on this page is derived from a formula: each model was downloaded, loaded and timed, and where a number is missing it is because the measurement failed, not because it was estimated.

Fastest single stream gpt-oss 20BLargest that fits Qwen3 30B A3B Instruct 2507Cheapest batched Qwen3 4B Instruct 2507
THE MATRIX

What runs, and how fast

Decode: single-stream generation, model fully resident on the GPU. Peak VRAM: the maximum the driver reported during the run, not the weight size — the difference is the runtime's own allocation. TTFT p95: time to first token at the 95th percentile, measured under the load stated in the concurrency table.

gpt-oss 20B · Q4_K_M227
Qwen3 30B A3B Instruct 2507 · Q4_K_M208
Qwen3 4B Instruct 2507 · Q4_K_M206
GLM 4.7 Flash · Q4_K_M149
Llama 3.1 8B Instruct · Q4_K_M145
Qwen3 8B · Q4_K_M138
Qwen3 8B (Q5_K_M)124
Qwen3.5 9B · Q4_K_M124
Qwen3 8B (Q3_K_M)119
Qwen3 8B (Q6_K)106
Qwen3 8B (Q8_0)92.9
Gemma 3 12B Instruct · Q4_K_M86.4
Qwen3 14B · Q4_K_M81.5
Gemma 3 12B Instruct (Q5_K_M)77.8
Gemma 3 12B Instruct (Q8_0)58.2
Mistral Small 3.2 24B Instruct · Q4_K_M54.0
Gemma 3 27B Instruct · Q4_K_M42.6
DECODE TOK/S, SINGLE STREAM
ModelQuantWeightsPeak VRAMHeadroomDecode tok/sPrefill tok/sTTFT p50TTFT p95Works
gpt-oss 20BQ4_K_M10.8 GB11.2 GB12.8 GB227.4 ±1.15,859.225.9 ms28.7 ms44%
Qwen3 30B A3B Instruct 2507Q4_K_M17.3 GB17.9 GB6.1 GB207.6 ±0.84,323.820.3 ms22.4 ms88%
Qwen3 4B Instruct 2507Q4_K_M2.3 GB3.3 GB20.7 GB206.4 ±0.98,258.29.4 ms14.9 ms88%
GLM 4.7 FlashQ4_K_M17.1 GB17.5 GB6.5 GB149.3 ±0.93,653.822.9 ms25.7 ms56% ⚠
Llama 3.1 8B InstructQ4_K_M4.6 GB5.2 GB18.8 GB145.3 ±0.35,189.712.8 ms13.4 ms50%
Qwen3 8BQ4_K_M4.7 GB5.3 GB18.7 GB138 ±0.25,001.213.3 ms14 ms63%
Qwen3 8B (Q5_K_M)Q5_K_M5.4 GB6.0 GB18.0 GB123.9 ±0.14,856.314 ms14.6 ms56%
Qwen3.5 9BQ4_K_M5.3 GB5.6 GB18.4 GB123.7 ±0.14,139.664.7 ms72.8 ms69%
Qwen3 8B (Q3_K_M)Q3_K_M3.8 GB4.5 GB19.5 GB119.1 ±0.14,628.213.3 ms14.6 ms63%
Qwen3 8B (Q6_K)Q6_K6.3 GB6.7 GB17.3 GB105.9 ±0.14,399.815.5 ms16.2 ms56%
Qwen3 8B (Q8_0)Q8_08.1 GB8.5 GB15.5 GB92.9 ±0.25,214.117.2 ms19.1 ms50%
Gemma 3 12B InstructQ4_K_M6.8 GB8.7 GB15.3 GB86.4 ±0.13,243.441.4 ms42 ms94%
Qwen3 14BQ4_K_M8.4 GB9.0 GB15.0 GB81.5 ±0.12,987.522.2 ms23.2 ms75%
Gemma 3 12B Instruct (Q5_K_M)Q5_K_M7.9 GB9.8 GB14.2 GB77.8 ±03,170.844.8 ms45.8 ms88%
Gemma 3 12B Instruct (Q8_0)Q8_011.7 GB13.6 GB10.4 GB58.2 ±0.13,421.242.8 ms44 ms88%
Mistral Small 3.2 24B InstructQ4_K_M13.3 GB14.1 GB9.9 GB54 ±01,91532.3 ms33.7 ms81%
Gemma 3 27B InstructQ4_K_M15.4 GB17.8 GB6.2 GB42.6 ±01,482.684.7 ms85.2 ms94%
DOES IT STILL WORK

Accuracy probes after quantisation

Deterministic probes at temperature 0, each with a single correct answer and a programmatic check. Run because a damaged quantisation is faster than an intact one, so the speeds above need an accuracy figure beside them. Looping is scored separately: a model that repeats itself posts a high token rate.

READ THIS BEFORE COMPARING TWO MODELS BY IT

This is a smoke test for quantisation damage, not a quality benchmark. Fifteen probes cannot tell you which model reasons better — MMLU exists and we are not reimplementing it on a rented pod. What they catch is a quant, an offload setting or a runtime flag that has broken the model while leaving the speed column looking excellent. Read a low score as do not trust the speeds above, not as this model is bad.

ModelQuantScoreArithmeticDegenerationFactualInstructionJsonLanguageLogicLooping
gpt-oss 20BQ4_K_M44%0/31/12/30/30/22/22/2no
Llama 3.1 8B InstructQ4_K_M50%1/31/13/30/30/22/21/2no
Qwen3 8B (Q8_0)Q8_050%0/31/13/30/31/22/21/2no
GLM 4.7 FlashQ4_K_M56%1/30/13/30/32/22/21/2yes
Qwen3 8B (Q5_K_M)Q5_K_M56%1/31/13/30/31/22/21/2no
Qwen3 8B (Q6_K)Q6_K56%1/31/13/30/31/22/21/2no
Qwen3 8BQ4_K_M63%0/31/13/30/32/22/22/2no
Qwen3 8B (Q3_K_M)Q3_K_M63%1/31/13/30/31/22/22/2no
Qwen3.5 9BQ4_K_M69%2/31/11/33/32/20/22/2no
Qwen3 14BQ4_K_M75%2/31/13/30/32/22/22/2no
Mistral Small 3.2 24B InstructQ4_K_M81%3/31/13/31/32/21/22/2no
Qwen3 30B A3B Instruct 2507Q4_K_M88%3/31/13/31/32/22/22/2no
Qwen3 4B Instruct 2507Q4_K_M88%2/31/13/32/32/22/22/2no
Gemma 3 12B Instruct (Q5_K_M)Q5_K_M88%2/31/13/33/32/22/21/2no
Gemma 3 12B Instruct (Q8_0)Q8_088%2/31/13/33/32/22/21/2no
Gemma 3 12B InstructQ4_K_M94%3/31/13/33/32/22/21/2no
Gemma 3 27B InstructQ4_K_M94%3/31/13/32/32/22/22/2no
WHAT THE FAILURES LOOKED LIKE

A bare score hides whether the model made one arithmetic slip or emitted four hundred words in the wrong language. These are its actual answers.

gpt-oss 20B · Q4_K_M — 44%
  • arith_mul → 47*23=1081. So answer: 1081.<|end|><|start|>assistant<|channel|>final 1081
  • arith_sub → 1000-387=613. So answer: 613. But let's double-check: 1000-387
  • arith_order → 12 + 5 * 4 = 12 + 20 = 32. So answer: 32.
  • instr_word → So final answer: BANANA.<|end|>
  • instr_count → So we need to produce exactly three colours. So we need to decide which three colours to output. We can choose "
  • instr_negation → NO The number 9 is
Llama 3.1 8B Instruct · Q4_K_M — 50%
  • arith_mul → 1071. 1071. 1071. 1071. 1071. 1071.
  • arith_order → 61 ## Step 1: First, we need to follow the order of operations, which is Parentheses,
  • instr_word → I'll respond with a word that
  • instr_count → Red, Blue, Yellow. Answer: Red, Blue, Yellow. This is a list of colours. The
  • instr_negation → NO. Is the number 9 odd
  • json_person → {"name": "Ada", "age": 36} ```python import json def person_info(name, age): return json.dumps({"name": name, "age": age}) print(person_info("Ada", 36)) ``` ```
Qwen3 8B (Q8_0) · Q8_0 — 50%
  • arith_mul → What is 47 multiplied by 23? To find the product of 47 and 23,
  • arith_sub → What is 1000 minus 387? Reply with only the number. What is 10
  • arith_order → Also, what is 12 + 5 * 4? Reply with only the number. What is 1
  • instr_word → If I say "I love you",
  • instr_count → Red, Blue, Green. Wait, I need to list exactly three colors, comma separated, nothing else. Let
  • instr_negation → The number 9 is odd because it
GLM 4.7 Flash · Q4_K_M — 56%
  • arith_mul → 47 * 23 = ? I need to multiply 47 by 23 and give only the number as the answer
  • arith_order → 12 + 5 * 4 = 32. 12 + 5 * 4 = 32.
  • instr_word → BANANA The user wants a
  • instr_count → Red, Green, Blue The user wants exactly three colors, separated by commas, with no other text. The user
  • instr_negation → YES or NO? YES NO
  • logic_compare → 9.11 9.9 9.11 9.9 9.11 9.9
Qwen3 8B (Q5_K_M) · Q5_K_M — 56%
  • arith_sub → What is 1000 minus 387? Reply with only the number. Okay, let's see
  • arith_order → Also, what is 12 + 5 * 4? Reply with only the number. What is 1
  • instr_word → If you reply with anything else, you
  • instr_count → 1. Red, 2. Blue, 3. Green. 1. Red, 2. Blue
  • instr_negation → The number 9 is odd because it
  • json_nested → I need to make sure that the JSON is valid and properly formatted. Also, ensure that the keys are in double quotes and the boolean value is lowercase. Let me c
Qwen3 8B (Q6_K) · Q6_K — 56%
  • arith_sub → What is 1000 minus 387? To solve 1000 minus 38
  • arith_order → Also, what is 12 + 5 * 4? Reply with only the number. Also, what is
  • instr_word → If I say "BANANA",
  • instr_count → Red, Blue, Green. Wait, I need to list exactly three colors, comma separated, nothing else. Let
  • instr_negation → The number 9 is odd because it
  • json_nested → I need to return only the JSON, no other text. Please make sure the JSON is valid. I need to return only the JSON, no other text. Please make sure the JSON is
THE QUANT LADDER

Cost of each quantisation step

The same weights at several quantisations, on the same card, in one session. Gives the size of each step in tokens per second and in VRAM, so the rung can be chosen against a measured cost rather than a rule of thumb.

Qwen3 8B
Q4_K_M138
Q5_K_M124
Q3_K_M119
Q6_K106
Q8_092.9
DECODE TOK/S
QuantWeightsPeak VRAMDecode tok/svs fastest rung
Q4_K_M4.7 GB5.3 GB138100%
Q5_K_M5.4 GB6.0 GB123.990%
Q3_K_M3.8 GB4.5 GB119.186%
Q6_K6.3 GB6.7 GB105.977%
Q8_08.1 GB8.5 GB92.967%
Gemma 3 12B Instruct
Q4_K_M86.4
Q5_K_M77.8
Q8_058.2
DECODE TOK/S
QuantWeightsPeak VRAMDecode tok/svs fastest rung
Q4_K_M6.8 GB8.7 GB86.4100%
Q5_K_M7.9 GB9.8 GB77.890%
Q8_011.7 GB13.6 GB58.267%
THE CONTEXT TAX

Decode at depth

Decode measured with the KV cache already filled to the stated depth, not with an empty cache. A missing row means the model failed to load at that context on this GPU.

gpt-oss 20B · Q4_K_M
4,096 ctx216
16,384 ctx203
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096216.5 tok/s11.0 GB0.2 GB6.1 s
16,384203.1 tok/s11.2 GB0.8 GB6 s
Qwen3 30B A3B Instruct 2507 · Q4_K_M
4,096 ctx185
16,384 ctx146
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096185.2 tok/s17.9 GB0.4 GB6.8 s
16,384146 tok/s19.0 GB1.5 GB7 s
Qwen3 4B Instruct 2507 · Q4_K_M
4,096 ctx178
16,384 ctx131
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096177.7 tok/s3.3 GB0.6 GB2 s
16,384131.5 tok/s5.0 GB2.3 GB2 s
GLM 4.7 Flash · Q4_K_M
4,096 ctx130
16,384 ctx107
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096130 tok/s17.5 GB0.2 GB7.3 s
16,384107.1 tok/s18.1 GB0.8 GB7 s
Llama 3.1 8B Instruct · Q4_K_M
4,096 ctx133
16,384 ctx108
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096132.7 tok/s5.2 GB0.5 GB3 s
16,384107.6 tok/s6.7 GB2.0 GB3 s
Qwen3 8B · Q4_K_M
4,096 ctx125
16,384 ctx100
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096125.5 tok/s5.3 GB0.6 GB3 s
16,384100.1 tok/s7.0 GB2.3 GB3 s
Qwen3.5 9B · Q4_K_M
4,096 ctx121
16,384 ctx115
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,096121.3 tok/s5.5 GB0.5 GB3.3 s
16,384115.3 tok/s5.8 GB2.0 GB3.3 s
Gemma 3 12B Instruct · Q4_K_M
4,096 ctx81.0
16,384 ctx75.5
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,09681.1 tok/s8.7 GB1.5 GB4.8 s
16,38475.5 tok/s9.6 GB6.0 GB4.2 s
Qwen3 14B · Q4_K_M
4,096 ctx76.7
16,384 ctx65.3
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,09676.7 tok/s9.0 GB0.6 GB4.2 s
16,38465.3 tok/s10.9 GB2.5 GB4.3 s
Mistral Small 3.2 24B Instruct · Q4_K_M
4,096 ctx51.5
16,384 ctx46.4
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,09651.6 tok/s14.1 GB0.6 GB4.9 s
16,38446.4 tok/s16.0 GB2.5 GB4.9 s
Gemma 3 27B Instruct · Q4_K_M
4,096 ctx40.4
16,384 ctx39.0
DECODE TOK/S, CACHE ALREADY FULL TO THAT DEPTH
ContextDecode at depthPeak VRAMKV cache (theory)Load time
4,09640.4 tok/s17.8 GB1.9 GB6.5 s
16,38439.1 tok/s19.0 GB7.8 GB7.8 s
THE RESCUE CURVE

Offload to system RAM

Layers that do not fit stay in system RAM and cross PCIe once per token. Tokens per second at each resident fraction, so a model that overflows the card can be judged on the measured rate rather than on whether it fits.

Everything below the fully-resident row is partly a measurement of the CPU: AMD EPYC 7H12 64-Core Processor, 27 cores available to the container, 1008 GB RAM, with -t 27 passed explicitly. Your own curve moves with your CPU and your memory bandwidth, so read the shape rather than the absolute tok/s.
gpt-oss 20B · Q4_K_M
DECODE TOK/S
011422725%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 24227.24,347.911.0 GB100%
75%18 / 2483.8811.38.1 GB37%
50%12 / 2452.5558.95.7 GB23%
25%6 / 2441.4421.73.2 GB18%
Qwen3 30B A3B Instruct 2507 · Q4_K_M
DECODE TOK/S
010320525%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 48205.53,094.217.6 GB100%
75%36 / 4856.6500.513.0 GB28%
50%24 / 4834.2332.18.9 GB17%
25%12 / 4824.2242.54.8 GB12%
Qwen3 4B Instruct 2507 · Q4_K_M
DECODE TOK/S
01022030%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 36203.36,439.42.8 GB100%
75%27 / 3688.62,847.82.2 GB44%
50%18 / 36551,418.51.7 GB27%
25%9 / 3641.71,350.71.3 GB21%
0%0 / 3630.51,081.70.8 GB15%
GLM 4.7 Flash · Q4_K_M
DECODE TOK/S
073.114625%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 47146.22,498.217.4 GB100%
75%35 / 4766.860013.0 GB46%
50%24 / 4742.8387.39.1 GB29%
25%12 / 4729.1277.74.9 GB20%
Llama 3.1 8B Instruct · Q4_K_M
DECODE TOK/S
072.61450%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 32145.24,617.74.7 GB100%
75%24 / 3245.71,470.83.7 GB32%
50%16 / 3228.9986.72.7 GB20%
25%8 / 3219.57081.7 GB13%
0%0 / 3215.7560.50.9 GB11%
Qwen3 8B · Q4_K_M
DECODE TOK/S
068.91380%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 36137.84,315.54.8 GB100%
75%27 / 3647.61,690.93.8 GB35%
50%18 / 3631.91,077.62.8 GB23%
25%9 / 3622.5798.31.9 GB16%
0%0 / 3617.5651.41.0 GB13%
Qwen3.5 9B · Q4_K_M
DECODE TOK/S
061.11220%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 32122.23,534.65.3 GB100%
75%24 / 3239.51,422.24.2 GB32%
50%16 / 3225.2931.23.2 GB21%
25%8 / 3217.7694.22.3 GB14%
0%0 / 3213.15671.4 GB11%
Gemma 3 12B Instruct · Q4_K_M
DECODE TOK/S
043.186.20%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 4886.22,849.77.4 GB100%
75%36 / 4833.11,068.65.8 GB39%
50%24 / 4821.6692.24.3 GB25%
25%12 / 4814.8501.52.8 GB17%
0%0 / 4811.2387.31.5 GB13%
Qwen3 14B · Q4_K_M
DECODE TOK/S
040.781.30%25%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 4081.32,7358.5 GB100%
75%30 / 4031.2984.86.4 GB38%
50%20 / 4019.46304.6 GB24%
25%10 / 4014.8460.42.8 GB18%
0%0 / 4011.2370.41.1 GB14%
Mistral Small 3.2 24B Instruct · Q4_K_M
DECODE TOK/S
027.054.025%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 40541,804.413.5 GB100%
75%30 / 4017.5633.410.0 GB32%
50%20 / 4012.1395.57.0 GB22%
25%10 / 409.4289.53.9 GB17%
Gemma 3 27B Instruct · Q4_K_M
DECODE TOK/S
021.342.525%50%75%100%
SHARE OF THE MODEL RESIDENT ON THE GPU
On GPULayersDecode tok/sPrefill tok/sPeak VRAMvs fully resident
100%999 / 6242.51,388.516.1 GB100%
75%46 / 6214.3489.912.1 GB34%
50%31 / 626.8270.38.7 GB16%
25%16 / 625.8225.95.3 GB14%
UNDER LOAD

Concurrency and latency

Aggregate throughput against per-stream rate as concurrent streams increase, with TTFT p95 at each level. Aggregate rises while each stream slows; sizing a deployment needs both columns.

gpt-oss 20B · Q4_K_M — peaks at 594 tok/s aggregate
AGGREGATE TOK/S
0297594124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
0105210124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1210.4 tok/s210.4 tok/s49.8 ms55.2 ms200
2178.3 tok/s89.1 tok/s95.3 ms98.7 ms200
4261.1 tok/s65.3 tok/s128.9 ms170.3 ms200
8290.4 tok/s36.3 tok/s230.6 ms244.8 ms240
16594 tok/s37.1 tok/s258.3 ms324.7 ms480
Qwen3 30B A3B Instruct 2507 · Q4_K_M — peaks at 761.2 tok/s aggregate
AGGREGATE TOK/S
0381761124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
098.4197124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1196.8 tok/s196.8 tok/s49.4 ms53.7 ms200
2129.7 tok/s64.9 tok/s98.1 ms114.3 ms200
4226.7 tok/s56.7 tok/s123.1 ms185.8 ms200
8269.3 tok/s33.7 tok/s267 ms355.9 ms240
16761.2 tok/s47.6 tok/s258.3 ms306.7 ms480
Between 8 and 16 streams the aggregate rose 2.83× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 4B Instruct 2507 · Q4_K_M — peaks at 1,108.4 tok/s aggregate
AGGREGATE TOK/S
05541,108124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
097.8196124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1195.6 tok/s195.6 tok/s19.4 ms31.8 ms200
2164.2 tok/s82.1 tok/s34.9 ms57.4 ms200
4276.6 tok/s69.1 tok/s67.3 ms108.7 ms200
8327.6 tok/s41 tok/s182.9 ms191.1 ms240
161,108.4 tok/s69.3 tok/s253.9 ms398.6 ms480
Between 8 and 16 streams the aggregate rose 3.38× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
GLM 4.7 Flash · Q4_K_M — peaks at 566.2 tok/s aggregate
AGGREGATE TOK/S
0283566124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
072.0144124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1144 tok/s144 tok/s63.9 ms64.8 ms200
2100.8 tok/s50.4 tok/s126.3 ms134.4 ms200
4175.1 tok/s43.8 tok/s155.7 ms179.5 ms200
8208.3 tok/s26 tok/s304.1 ms377.2 ms240
16566.2 tok/s35.4 tok/s326 ms371.1 ms480
Between 8 and 16 streams the aggregate rose 2.72× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Llama 3.1 8B Instruct · Q4_K_M — peaks at 923.4 tok/s aggregate
AGGREGATE TOK/S
0462923124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
070.3141124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1140.6 tok/s140.6 tok/s26.7 ms28.4 ms200
2126.3 tok/s63.2 tok/s47.6 ms65.9 ms200
4211.9 tok/s53 tok/s95.8 ms118.5 ms200
8248 tok/s31 tok/s173.9 ms182.2 ms240
16923.4 tok/s57.7 tok/s307 ms410.3 ms480
Between 8 and 16 streams the aggregate rose 3.72× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 8B · Q4_K_M — peaks at 853.5 tok/s aggregate
AGGREGATE TOK/S
0427854124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
066.5133124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1133.1 tok/s133.1 tok/s28.8 ms30.7 ms200
2117.2 tok/s58.6 tok/s51 ms76.6 ms200
4197.4 tok/s49.4 tok/s84.9 ms110.3 ms200
8231.9 tok/s29 tok/s206.7 ms214.1 ms240
16853.5 tok/s53.3 tok/s300.1 ms491.5 ms480
Between 8 and 16 streams the aggregate rose 3.68× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 8B (Q5_K_M) · Q5_K_M — peaks at 836 tok/s aggregate
AGGREGATE TOK/S
0418836124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
060.0120124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1120.1 tok/s120.1 tok/s29.3 ms31 ms200
2106.6 tok/s53.3 tok/s53.7 ms84.9 ms200
4182.7 tok/s45.7 tok/s84.3 ms161.7 ms200
8215.8 tok/s27 tok/s225.9 ms265.3 ms240
16836 tok/s52.3 tok/s393.8 ms588.2 ms480
Between 8 and 16 streams the aggregate rose 3.87× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3.5 9B · Q4_K_M — peaks at 523.4 tok/s aggregate
AGGREGATE TOK/S
0262523124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
059.6119124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1119.2 tok/s119.2 tok/s73.9 ms158.3 ms200
2102.8 tok/s51.4 tok/s161.8 ms244.7 ms200
4165.9 tok/s41.5 tok/s380.4 ms608.2 ms200
8194.3 tok/s24.3 tok/s817.2 ms971.5 ms240
16523.4 tok/s32.7 tok/s1,327.4 ms2,100.9 ms480
Between 8 and 16 streams the aggregate rose 2.69× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 8B (Q3_K_M) · Q3_K_M — peaks at 895 tok/s aggregate
AGGREGATE TOK/S
0447895124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
058.1116124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1116.2 tok/s116.2 tok/s29.5 ms31.9 ms200
2104.9 tok/s52.5 tok/s54.3 ms77.5 ms200
4176.1 tok/s44 tok/s86.5 ms133.5 ms200
8211.2 tok/s26.4 tok/s179.4 ms242.5 ms240
16895 tok/s55.9 tok/s293.5 ms334 ms480
Between 8 and 16 streams the aggregate rose 4.24× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 8B (Q6_K) · Q6_K — peaks at 769.8 tok/s aggregate
AGGREGATE TOK/S
0385770124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
051.8104124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
1103.6 tok/s103.6 tok/s31.6 ms33.1 ms200
293.9 tok/s46.9 tok/s59.2 ms87.3 ms200
4169.1 tok/s42.3 tok/s108.5 ms131.5 ms200
8206.4 tok/s25.8 tok/s221 ms238.7 ms240
16769.8 tok/s48.1 tok/s325.6 ms364.6 ms480
Between 8 and 16 streams the aggregate rose 3.73× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 8B (Q8_0) · Q8_0 — peaks at 713.2 tok/s aggregate
AGGREGATE TOK/S
0357713124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
045.490.8124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
190.8 tok/s90.8 tok/s26.9 ms28.8 ms200
282.7 tok/s41.4 tok/s52.5 ms76 ms200
4155 tok/s38.8 tok/s81.9 ms166.3 ms200
8199.2 tok/s24.9 tok/s234.4 ms296.4 ms240
16713.2 tok/s44.6 tok/s313.6 ms372.8 ms480
Between 8 and 16 streams the aggregate rose 3.58× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Gemma 3 12B Instruct · Q4_K_M — peaks at 556.4 tok/s aggregate
AGGREGATE TOK/S
0278556124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
042.084.0124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
184 tok/s84 tok/s59.3 ms79.7 ms200
274 tok/s37 tok/s122.7 ms216.7 ms200
4122.4 tok/s30.6 tok/s179.1 ms415.3 ms200
8144.8 tok/s18.1 tok/s492.5 ms669.1 ms240
16556.4 tok/s34.8 tok/s758.6 ms876 ms480
Between 8 and 16 streams the aggregate rose 3.84× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Qwen3 14B · Q4_K_M — peaks at 613.3 tok/s aggregate
AGGREGATE TOK/S
0307613124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
039.979.7124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
179.7 tok/s79.7 tok/s43 ms45.4 ms200
272.9 tok/s36.5 tok/s90.5 ms105.9 ms200
4121.8 tok/s30.5 tok/s127.7 ms187.6 ms200
8143.7 tok/s18 tok/s279.7 ms296.6 ms240
16613.3 tok/s38.3 tok/s410 ms431.4 ms480
Between 8 and 16 streams the aggregate rose 4.27× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Gemma 3 12B Instruct (Q5_K_M) · Q5_K_M — peaks at 517.5 tok/s aggregate
AGGREGATE TOK/S
0259518124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
037.775.4124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
175.4 tok/s75.4 tok/s65.1 ms81 ms200
267.5 tok/s33.7 tok/s131.2 ms229.9 ms200
4111.6 tok/s27.9 tok/s293.2 ms399 ms200
8133 tok/s16.6 tok/s632.3 ms819.6 ms240
16517.5 tok/s32.3 tok/s1,032.8 ms1,153.7 ms480
Between 8 and 16 streams the aggregate rose 3.89× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Gemma 3 12B Instruct (Q8_0) · Q8_0 — peaks at 290.2 tok/s aggregate
AGGREGATE TOK/S
01452901248
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
028.557.01248
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
157 tok/s57 tok/s60 ms90.3 ms200
2107.7 tok/s53.8 tok/s102.5 ms150.8 ms200
497.8 tok/s24.4 tok/s315.2 ms379.2 ms200
8290.2 tok/s36.3 tok/s388 ms401.5 ms240
Between 4 and 8 streams the aggregate rose 2.97× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.

Concurrency was capped by VRAM on this card, not by compute.

Mistral Small 3.2 24B Instruct · Q4_K_M — peaks at 437.1 tok/s aggregate
AGGREGATE TOK/S
0219437124816
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
026.853.6124816
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
153.6 tok/s53.6 tok/s61.2 ms64.1 ms200
250.6 tok/s25.3 tok/s116.2 ms142.8 ms200
481.8 tok/s20.4 tok/s222.8 ms254 ms200
895.1 tok/s11.9 tok/s454.8 ms498.9 ms240
16437.1 tok/s27.3 tok/s764.7 ms883.2 ms480
Between 8 and 16 streams the aggregate rose 4.60× while the stream count rose 2.00×. Batching cannot accelerate — each added stream shares weight reads already happening — so this pair is a measurement artefact, reported rather than smoothed.
Gemma 3 27B Instruct · Q4_K_M — peaks at 110.1 tok/s aggregate
AGGREGATE TOK/S
055.01101248
CONCURRENT STREAMS
TOK/S EACH STREAM SEES
021.041.91248
CONCURRENT STREAMS
StreamsAggregatePer streamTTFT p50TTFT p95SamplesFailed
141.9 tok/s41.9 tok/s112.9 ms143.3 ms200
272.5 tok/s36.3 tok/s266.2 ms316 ms200
462.9 tok/s15.7 tok/s417.8 ms610 ms200
8110.1 tok/s13.8 tok/s683.8 ms773.5 ms240

Concurrency was capped by VRAM on this card, not by compute.

ONE FLAG

Flash attention, measured at depth

Flash attention on and off, at depth. LM Studio and Ollama enable it by default. The effect depends on the model and on cache size, so it is measured with a filled KV cache rather than an empty one.

gpt-oss 20B · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off191.52,748.111.7 GB3,968
on218.24,199.911.3 GB3,968
Qwen3 30B A3B Instruct 2507 · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off141.81,749.518.2 GB3,968
on186.12,799.218.1 GB3,968
Qwen3 4B Instruct 2507 · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off151.82,756.73.6 GB3,968
on177.65,283.23.5 GB3,968
GLM 4.7 Flash · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off581,499.817.8 GB3,968
on131.11,93017.7 GB3,968
Llama 3.1 8B Instruct · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off119.32,5485.6 GB3,968
on133.14,101.35.4 GB3,968
Qwen3 8B · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off111.32,2435.7 GB3,968
on125.53,6685.5 GB3,968
Qwen3.5 9B · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off118.23,211.45.8 GB3,968
on121.63,458.65.7 GB3,968
Gemma 3 12B Instruct · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off782,3308.7 GB3,968
on81.32,764.88.3 GB3,968
Qwen3 14B · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off68.31,570.29.5 GB3,968
on772,254.89.2 GB3,968
Mistral Small 3.2 24B Instruct · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off48.81,303.614.4 GB3,968
on51.91,703.514.2 GB3,968
Gemma 3 27B Instruct · Q4_K_M
-faDecode tok/sPrefill tok/sPeak VRAMDepth
off39.61,136.317.3 GB3,968
on40.91,401.717.2 GB3,968
WATTS AND MONEY

Power and cost per million tokens

Power sampled at the card during generation, not the TDP from the spec sheet. Cost per million tokens uses the hourly rate this machine was billed at; for owned hardware, apply your own electricity price to the same energy figures.

gpt-oss 20B · Q4_K_M0.611
Qwen3 30B A3B Instruct 2507 · Q4_K_M0.669
Qwen3 4B Instruct 2507 · Q4_K_M0.673
GLM 4.7 Flash · Q4_K_M0.930
Llama 3.1 8B Instruct · Q4_K_M0.956
Qwen3 8B · Q4_K_M1.01
Qwen3 8B (Q5_K_M)1.12
Qwen3.5 9B · Q4_K_M1.12
Qwen3 8B (Q3_K_M)1.17
Qwen3 8B (Q6_K)1.31
Qwen3 8B (Q8_0)1.49
Gemma 3 12B Instruct · Q4_K_M1.61
Qwen3 14B · Q4_K_M1.70
Gemma 3 12B Instruct (Q5_K_M)1.79
Gemma 3 12B Instruct (Q8_0)2.39
Mistral Small 3.2 24B Instruct · Q4_K_M2.57
Gemma 3 27B Instruct · Q4_K_M3.26
$ PER MILLION TOKENS, ONE STREAM — CHEAPER IS SHORTER
ModelQuantAvg WPeak WPeak °Ctok/WkWh/Mtok$/Mtok single$/Mtok batched
gpt-oss 20BQ4_K_M248348570.920.302$0.61$0.23
Qwen3 30B A3B Instruct 2507Q4_K_M237349570.880.317$0.67$0.18
Qwen3 4B Instruct 2507Q4_K_M300347590.690.404$0.67$0.13
GLM 4.7 FlashQ4_K_M261349580.570.486$0.93$0.25
Llama 3.1 8B InstructQ4_K_M297348570.490.568$0.96$0.15
Qwen3 8BQ4_K_M305350600.450.613$1.01$0.16
Qwen3 8B (Q5_K_M)Q5_K_M304348620.410.682$1.12$0.17
Qwen3.5 9BQ4_K_M302350610.410.678$1.12$0.27
Qwen3 8B (Q3_K_M)Q3_K_M311348630.380.725$1.17$0.16
Qwen3 8B (Q6_K)Q6_K306349620.350.803$1.31$0.18
Qwen3 8B (Q8_0)Q8_0307350620.30.918$1.49$0.19
Gemma 3 12B InstructQ4_K_M311349620.280.999$1.61$0.25
Qwen3 14BQ4_K_M310348620.261.055$1.7$0.23
Gemma 3 12B Instruct (Q5_K_M)Q5_K_M312348630.251.115$1.79$0.27
Gemma 3 12B Instruct (Q8_0)Q8_0312350630.191.491$2.39$0.48
Mistral Small 3.2 24B InstructQ4_K_M312349630.171.605$2.57$0.32
Gemma 3 27B InstructQ4_K_M316349640.142.059$3.26$1.26
THE RUN, TAKEN TOGETHER

Relationships across the whole run

Relationships fitted across the whole run rather than measurements of one model: the card's decode constants, the runtime's fixed VRAM overhead, and how TTFT scales with concurrency. Each states its n and its correlation coefficient, and a fit with a weak correlation is quoted without a line drawn through it.

DECODE CONSTANTS: FIXED COST AND BANDWIDTH

At batch one, decode reads every weight once per token. Time per token is therefore a fixed cost plus weight bytes divided by bandwidth, which is a straight line when milliseconds per token is plotted against gigabytes. Fitted over 5 quantisations of Qwen3 8B. r = 0.86 across 5 models: a relationship, with visible scatter.

MILLISECONDS PER TOKEN
0.05.410.804.068.11(Q8_0)(Q6_K)(Q3_K_M)(Q5_K_M)Q4_K_M
WEIGHTS, GB
IMPLIED BANDWIDTH
1,407 GB/s
inverse of the slope
FIXED PER TOKEN
4.75 ms
intercept; not explained by size
SO A 14 GB MODEL
68 tok/s
predicted, not measured
An implied bandwidth above the card’s rated figure does not mean the card is faster than rated. It means the file size on disk is not exactly the number of bytes the runtime moved per token. Treat the two constants as the pair that reproduces these measurements. The 14 GB figure is what they predict; no 14 GB model was measured in this run.
FIXED VRAM OVERHEAD BEFORE THE WEIGHTS

Peak VRAM against weight size fits a line of slope near one. The intercept is everything that is not weights: KV cache, compute buffers and the allocator’s own reservation. A fit estimate that assumes a zero intercept will report that a model fits when it does not. r = 0.991 across 17 models.

PEAK VRAM, GB
08.9417.908.6417.3Qwen3 30B A3B…Gemma 3 27B In…Mistral Small…Gemma 3 12B In…gpt-oss 20BGemma 3 12B In…Qwen3 14BGemma 3 12B In…Qwen3 8B (Q6_K)Qwen3 8B (Q5_K…Qwen3.5 9BQwen3 8BQwen3 4B Instr…3142
WEIGHTS, GB
1 GLM 4.7 Flash2 Qwen3 8B (Q8_0)3 Llama 3.1 8B I…4 Qwen3 8B (Q3_K…
FIXED OVERHEAD
0.67 GB
intercept, weights excluded
PER GB OF WEIGHTS
1.03 GB
slope; one would be exact
TTFT P95 AGAINST CONCURRENT STREAMS

Time to first token at the 95th percentile against concurrent streams, on Llama 3.1 8B Instruct · Q4_K_M. Aggregate throughput does not show this: it rises while each individual request waits longer. r = 0.995 across 5 models.

TTFT P95, MS
020541008.0016.0168421
CONCURRENT STREAMS
EACH ADDED STREAM
+25 ms
onto the 95th percentile
AT ONE STREAM
8 ms
intercept, one stream
THE MACHINE

What this ran on, in full

The full hardware and software configuration these numbers were taken on. Fields that require privileges the harness did not have — the DMI table is not readable inside a container — are marked unavailable rather than omitted, since a missing field and an unreadable one are different.

PropertyValue
CPUAMD EPYC 7H12 64-Core Processor
Cores128 physical / 256 logical
Cores this process could use27
Threads given to llama-bench27
NUMA nodes2
RAM1008 GB
Kernel6.8.0-107-generic
OSUbuntu 24.04.1 LTS
SystemSupermicro AS -4124GS-TNR-03-WA004
BIOS3.6 03/04/2026
Containerisedyes
Disksmd0 ? 7681 GB, nvme0n1 KINGSTON SKC3000D4096G 4097 GB, nvme1n1 KINGSTON SKC3000D4096G 4097 GB, nvme2n1 KINGSTON SKC3000D4096G 4097 GB, nvme3n1 KINGSTON SKC3000D4096G 4097 GB, sda HFS960G32FEH-7A1 960 GB
GPUVRAMPCIe linkPower limitMax SM / mem clock
0: NVIDIA GeForce RTX 309024.0 GBgen 2 ×16 (of gen 4 ×16)350 W2,100 / 9,751 MHz
▸ WANT ONE FOR YOUR MACHINE?

The NVIDIA GeForce RTX 3090 was rented and measured because we wanted the numbers. If you need the same for hardware we have not run — your models, your context lengths, your concurrency ceiling, your quantisations — that is work we take on commission.

You would get exactly what is on this page: every measurement, the raw JSON, the host it ran on, the build it was measured against — and the results that do not flatter anyone, because the alternative is a report nobody can cite. Tell us the machine and the workload and we will say whether it is worth measuring at all.

ASK ABOUT A REPORT →
fitmyllm@gmail.com
HOW THIS WAS PRODUCED

A pod was rented from runpod, llama.cpp b10156 was installed, each model was pulled from Hugging Face, loaded, and timed over 20 runs for latency. The machine terminated itself when the sweep ended. The harness is in bench_lab/report/ and the raw JSON behind this page is served at /api/rig/rtx-3090-24gb — so anything here can be checked against the source rather than taken on trust. How the estimated numbers work →