▸ AMD· CDNA 3.0

AMD Radeon Instinct MI300X

Running LLMs on the Radeon Instinct MI300X — the long read: which models fit at which quantisation, and the settings worth changing. · Or what else $15,000 buys

VRAM

192 GB

FLAGSHIP

BANDWIDTH

10300

GB/S

MODELS Q4

407/449

91%

7B Q4 SPEED

~1308

BLAZING

▸ MODEL COVERAGE @ Q491% OF ALL

▸ ESTIMATED SPEED· BY MODEL SIZE @ Q4

Average speeds at Q4 quantization. Actual performance varies by model architecture and context length.

~3052

TOK/S

~1308

TOK/S

14B

~654

TOK/S

32B

~286

TOK/S

70B

~131

TOK/S

▸ MEASURED RIG REPORTS

We rent the machine and time every model on it: decode, VRAM peak, concurrency, watts and cost per million tokens. Yours may already be one of them — and two are free to read in full.

SEE THE REPORTS →

▸ SPECIFICATIONS

VRAM

192 GB

BANDWIDTH

10300 GB/s

FP16 COMPUTE

653.7 TFLOPS

TDP

750W

MEMORY

HBM3

ARCHITECTURE

CDNA 3.0

STREAM PROCESSORS

19,456

COMPUTE UNITS

152

PCIE

Gen 5 x16

MSRP

$15,000

407

FAST MODELS · >30 TOK/S

Real-time chat speed

407

USABLE · >10 TOK/S

Comfortable for all tasks

407

TOTAL COMPATIBLE

Fit in VRAM at Q4

▸ RENT IT IN THE CLOUD

Buying Radeon Instinct MI300X costs $15–$40k and isn’t practical for most teams. Spin one up by the hour instead:

Rent on Vast.ai →Or RunPod →

Spin up in ~60s. Pay by the second. Cancel anytime.

Some links are affiliate links — we may earn a small commission at no extra cost to you. This helps keep FitMyLLM free and independent.

▸ COMPATIBLE MODELS· 407

Falcon-H1R Tiny 90M0.09B

FALCON·256K CTX· CHAT· REASONING

nomic-embed-text-v1.5 100M0.14B

EMBEDDING·8K CTX· CHAT

SMOLLM·8K CTX· CHAT· VISION

Snowflake Arctic Embed M v2.00.305B

EMBEDDING·8K CTX· EMBEDDING· MULTILINGUAL

bge-large-en-v1.5 335M0.335B

EMBEDDING·1K CTX· CHAT

mxbai-embed-large-v10.335B

EMBEDDING·1K CTX· EMBEDDING

Snowflake Arctic Embed L0.335B

EMBEDDING·1K CTX· EMBEDDING

LFM·125K CTX· CHAT· TOOL_USE

GPT-2 Medium 345M0.38B

SMOLLM·8K CTX· CHAT· VISION

FALCON·128K CTX· CHAT

EMBEDDING·8K CTX· EMBEDDING

Snowflake Arctic Embed L v2.00.568B

EMBEDDING·8K CTX· EMBEDDING· MULTILINGUAL

Qwen3-Embedding 0.6B0.6B

EMBEDDING·32K CTX· EMBEDDING· MULTILINGUAL

Falcon-H1R Tiny 0.6B0.6B

FALCON·32K CTX· CHAT· REASONING

Falcon Perception 0.6B0.6B

FALCON·4K CTX· VISION

LFM·125K CTX· CHAT· TOOL_USE

QWEN·32K CTX· CHAT· REASONING

GPT-2 Large 774M0.81B

QWEN·256K CTX· CHAT· CODING· MULTILINGUAL

OTHER·32K CTX· CHAT· VISION

MINICPM·128K CTX· CHAT· REASONING· MULTILINGUAL

LFM2.5-1.2B-Thinking1.2B

LFM·122K CTX· CHAT· REASONING· TOOL_USE

LFM·125K CTX· CHAT· TOOL_USE· MULTILINGUAL

OTHER·256K CTX· CHAT· VISION

DeepSeek Coder 1.3B1.35B

DEEPSEEK·16K CTX· CODING

PHI·2K CTX· CHAT· CODING

LFM·125K CTX· CHAT· TOOL_USE· MULTILINGUAL

▸ NEXT STEP

Get personalized recommendations.

See ranked models with benchmark scores, run commands, and precise speed estimates for your Radeon Instinct MI300X.

▸Find models for Radeon Instinct MI300X Compare with other GPUs

WHAT THIS CARD IS WORTH

Radeon Instinct MI300X holds 407 of the models in our catalogue and is, in practice, a IQ4_XS card — the largest it takes is DeepSeek V4 Flash 0731 at IQ4_XS.

TOKENS/SEC PER $100

7.3

8B at Q4_K_M, so cards compare like for like

VRAM PER $100

1.28 GB

what memory costs on this card

THE BIGGEST IT TAKES

DeepSeek V4 Flash 0731304.18B · IQ4_XS172.4 GB754 tok/sEST

ERNIE-4.5-300B-A47B-Paddle300.5B · IQ4_XS170.9 GB40 tok/sEST

ERNIE 4.5 300B-A47B300B · IQ4_XS170.6 GB255 tok/sEST

Hunyuan Hy3-preview295B · IQ4_XS168.2 GB511 tok/sEST

DeepSeek V4 Flash290.94B · Q4_K_S172.7 GB726 tok/sEST

DeepSeek V2 236B236B · Q5_K_M170.9 GB418 tok/sEST

DeepSeek-V2.5 236B236B · Q5_K_M170.9 GB418 tok/sEST

DeepSeek-Coder-V2 236B236B · Q5_K_M170.9 GB418 tok/sEST

AGAINST CARDS OF SIMILAR MEMORY

Radeon Instinct MI300A192 GB$12,0009.2 tok/s per $100

Radeon Instinct MI308X192 GB$12,0009.2 tok/s per $100

Instinct MI300X192 GB$15,0003.8 tok/s per $100

M4 Ultra (192GB)144 GB$7,4991.2 tok/s per $100

Neighbours in memory rather than in price: memory decides whether a card can do the job at all, so two cards of the same size at different prices is the comparison you are making. Ordered by memory, then bandwidth — not by the value column, which is worked out from bandwidth and price alone and therefore rewards a cheap card whatever its software stack does to that bandwidth in practice. Read it as one input, not as a ranking.

▸ DEVICE UNDER TEST

AMD Radeon Instinct MI300X — 192 GB VRAM.

▸ RADEON INSTINCT MI300X SPEC

BRAND: AMD
VRAM: 192 GB HBM3
BANDWIDTH: 10300 GB/s
FP16 COMPUTE: 653.7 TFLOPS
FP32 COMPUTE: 81.7 TFLOPS
STREAM PROCESSORS: 19,456
TDP: 750 W
ARCHITECTURE: CDNA 3.0
MSRP: $15000

▸ AI CAPABILITY

407/ 449 models @ Q4

With 192 GB VRAM and 10300 GB/s bandwidth, this GPU handles models up to 236B parameters.

Speed ≈ bandwidth / model_size × efficiency. A 7B model at Q4 runs at ~1177 tok/s.

RENT IT — LIVE PRICES

Checking what the clouds are charging…

§ 01TOP MODELS FOR RADEON INSTINCT MI300X

407 FIT · SHOWING 20

MODEL	SIZE	VRAM Q4	TOK/S	AVG
DeepSeek V2 236B	236B	144.7 GB	436	40.4
DeepSeek-V2.5 236B	236B	144.7 GB	436	64.0
DeepSeek-Coder-V2 236B	236B	144.7 GB	436	61.8
Qwen3-235B-A22B	235.1B	144.2 GB	416	55.9
Qwen3-235B-A22B Instruct 2507	235.1B	144.2 GB	416	51.9
Qwen3-VL 235B A22B Instruct	235B	144.1 GB	416	44.3
MiniMax-M2.7	230B	141.1 GB	916	58.1
MiniMax-M2.5 228.7B	228.7B	140.3 GB	436	62.6
Command A+	218B	133.7 GB	366	41.8
Step 3.7 Flash	201.37B	123.6 GB	832	53.8
Falcon 180B	180B	110.5 GB	51	51.3
bloom 176.2B	176.2B	108.2 GB	52	15.0
dots.llm1.inst 142.8B	142.8B	87.8 GB	64	—
WizardLM 2 8x22B	141B	86.7 GB	235	42.4
Mixtral-8x22B	140.6B	86.4 GB	234	31.9
DBRX 132B	132B	81.2 GB	254	46.3
Mistral Medium 3.5	128B	78.7 GB	72	46.6
Pixtral Large 124B	124B	76.3 GB	74	39.3
Nemotron 3 Super 120B-A12B	123.61B	76.0 GB	763	53.2
Mistral-Large 123B	123B	75.7 GB	74	33.5