▸ NVIDIA· AMPERE

NVIDIA A100 SXM 80GB

Running LLMs on the A100 SXM 80GB — the long read: which models fit at which quantisation, and the settings worth changing. · Or what else $10,000 buys

VRAM

80 GB

FLAGSHIP

BANDWIDTH

2039

GB/S

MODELS Q4

382/449

85%

7B Q4 SPEED

~233

BLAZING

▸ MODEL COVERAGE @ Q485% OF ALL

▸ ESTIMATED SPEED· BY MODEL SIZE @ Q4

Average speeds at Q4 quantization. Actual performance varies by model architecture and context length.

~544

TOK/S

~233

TOK/S

14B

~117

TOK/S

32B

~51

TOK/S

70B

~23

TOK/S

▸ MEASURED RIG REPORTS

We rent the machine and time every model on it: decode, VRAM peak, concurrency, watts and cost per million tokens. Yours may already be one of them — and two are free to read in full.

SEE THE REPORTS →

▸ SPECIFICATIONS

VRAM

80 GB

BANDWIDTH

2039 GB/s

FP16 COMPUTE

312 TFLOPS

TDP

400W

MEMORY

HBM2e

ARCHITECTURE

Ampere

CUDA CORES

6,912

TENSOR CORES

432

PCIE

Gen 4 x16

MSRP

$10,000

351

FAST MODELS · >30 TOK/S

Real-time chat speed

382

USABLE · >10 TOK/S

Comfortable for all tasks

382

TOTAL COMPATIBLE

Fit in VRAM at Q4

▸ RENT IT IN THE CLOUD

Buying A100 SXM 80GB costs $15–$40k and isn’t practical for most teams. Spin one up by the hour instead:

Rent on Vast.ai →Or RunPod →

Spin up in ~60s. Pay by the second. Cancel anytime.

Some links are affiliate links — we may earn a small commission at no extra cost to you. This helps keep FitMyLLM free and independent.

▸ COMPATIBLE MODELS· 382

Falcon-H1R Tiny 90M0.09B

FALCON·256K CTX· CHAT· REASONING

nomic-embed-text-v1.5 100M0.14B

EMBEDDING·8K CTX· CHAT

SMOLLM·8K CTX· CHAT· VISION

Snowflake Arctic Embed M v2.00.305B

EMBEDDING·8K CTX· EMBEDDING· MULTILINGUAL

bge-large-en-v1.5 335M0.335B

EMBEDDING·1K CTX· CHAT

mxbai-embed-large-v10.335B

EMBEDDING·1K CTX· EMBEDDING

Snowflake Arctic Embed L0.335B

EMBEDDING·1K CTX· EMBEDDING

LFM·125K CTX· CHAT· TOOL_USE

GPT-2 Medium 345M0.38B

SMOLLM·8K CTX· CHAT· VISION

FALCON·128K CTX· CHAT

EMBEDDING·8K CTX· EMBEDDING

Snowflake Arctic Embed L v2.00.568B

EMBEDDING·8K CTX· EMBEDDING· MULTILINGUAL

Qwen3-Embedding 0.6B0.6B

EMBEDDING·32K CTX· EMBEDDING· MULTILINGUAL

Falcon-H1R Tiny 0.6B0.6B

FALCON·32K CTX· CHAT· REASONING

Falcon Perception 0.6B0.6B

FALCON·4K CTX· VISION

LFM·125K CTX· CHAT· TOOL_USE

QWEN·32K CTX· CHAT· REASONING

GPT-2 Large 774M0.81B

QWEN·256K CTX· CHAT· CODING· MULTILINGUAL

OTHER·32K CTX· CHAT· VISION

MINICPM·128K CTX· CHAT· REASONING· MULTILINGUAL

LFM2.5-1.2B-Thinking1.2B

LFM·122K CTX· CHAT· REASONING· TOOL_USE

LFM·125K CTX· CHAT· TOOL_USE· MULTILINGUAL

OTHER·256K CTX· CHAT· VISION

DeepSeek Coder 1.3B1.35B

DEEPSEEK·16K CTX· CODING

PHI·2K CTX· CHAT· CODING

LFM·125K CTX· CHAT· TOOL_USE· MULTILINGUAL

▸ NEXT STEP

Get personalized recommendations.

See ranked models with benchmark scores, run commands, and precise speed estimates for your A100 SXM 80GB.

▸Find models for A100 SXM 80GB Compare with other GPUs

WHAT THIS CARD IS WORTH

A100 SXM 80GB holds 382 of the models in our catalogue and is, in practice, a IQ4_XS card — the largest it takes is Nemotron 3 Super 120B-A12B at IQ4_XS.

TOKENS/SEC PER $100

0.4

8B at Q4_K_M, so cards compare like for like

VRAM PER $100

0.80 GB

what memory costs on this card

THE BIGGEST IT TAKES

Nemotron 3 Super 120B-A12B123.61B · IQ4_XS71.8 GB27 tok/sEST

Qwen 3.5 122B A10B122B · IQ4_XS70.6 GB33 tok/sEST

Nemotron 3 Super 120B120B · IQ4_XS70.7 GB27 tok/sEST

Mistral Small 4 119B119B · IQ4_XS68.9 GB51 tok/sEST

GPT-OSS 120B117B · Q4_K_S70.9 GB62 tok/sEST

Command A 111B111B · Q4_K_M71.3 GB3 tok/sEST

GLM 4.5 Air110B · Q4_K_M70.5 GB25 tok/sEST

Qwen 1.5 110B110B · Q4_K_M71.0 GB3 tok/sEST

AGAINST CARDS OF SIMILAR MEMORY

M1 Ultra (128GB)96 GB$4,9991.4 tok/s per $100

M2 Ultra (128GB)96 GB$3,9991.8 tok/s per $100

M4 Max (128GB)96 GB$3,9991.4 tok/s per $100

H100 PCIe 96 GB96 GB$25,0000.1 tok/s per $100

A100 SXM 80GB80 GB$10,0000.4 tok/s per $100

Neighbours in memory rather than in price: memory decides whether a card can do the job at all, so two cards of the same size at different prices is the comparison you are making. Ordered by memory, then bandwidth — not by the value column, which is worked out from bandwidth and price alone and therefore rewards a cheap card whatever its software stack does to that bandwidth in practice. Read it as one input, not as a ranking.

▸ DEVICE UNDER TEST

NVIDIA A100 SXM 80GB — 80 GB VRAM.

▸ A100 SXM 80GB SPEC

BRAND: NVIDIA
VRAM: 80 GB HBM2e
BANDWIDTH: 2039 GB/s
FP16 COMPUTE: 312 TFLOPS
FP32 COMPUTE: 19.5 TFLOPS
CUDA CORES: 6,912
TENSOR CORES: 432
TDP: 400 W
ARCHITECTURE: Ampere
MSRP: $10000

▸ AI CAPABILITY

382/ 449 models @ Q4

With 80 GB VRAM and 2039 GB/s bandwidth, this GPU handles models up to 111B parameters.

Speed ≈ bandwidth / model_size × efficiency. A 7B model at Q4 runs at ~233 tok/s.

RENT IT — LIVE PRICES

Checking what the clouds are charging…

§ 01TOP MODELS FOR A100 SXM 80GB

382 FIT · SHOWING 20

MODEL	SIZE	VRAM Q4	TOK/S	AVG
Command A 111B	111B	68.3 GB	15	27.6
GLM 4.5 Air	110B	67.7 GB	136	51.0
Qwen 1.5 110B	110B	67.7 GB	15	33.4
Llama 4 Scout 17B-16E	109B	67.1 GB	96	33.9
Cogito v2 109B MoE	109B	67.1 GB	96	—
Ling 2.6 Flash	107.49B	66.2 GB	220	36.8
Sarvam 105B	105B	64.7 GB	16	48.0
Command-R+ 104B	104B	64.1 GB	16	52.7
Llama-3.2-90B-Vision-Instruct	90B	55.5 GB	18	48.5
Hunyuan A13B	80B	49.4 GB	125	81.1
Qwen3-Coder-Next	80B	49.4 GB	544	43.0
Qwen3-Next 80B A3B	80B	49.4 GB	544	49.0
NVLM-D 72B	79.38B	49.0 GB	21	48.7
InternVL3 78B	78B	48.2 GB	21	80.6
Qwen2.5-72B	72.7B	44.9 GB	22	39.7
Qwen2-VL 72B	72.7B	44.9 GB	22	55.5
Qwen 1.5 72B	72B	44.5 GB	23	49.7
Qwen2 Math 72B	72B	44.5 GB	23	49.7
Molmo 72B	72B	44.5 GB	23	54.1
DeepSeek R1 Distill Llama 70B	70.6B	43.6 GB	23	42.4