Microsoft/Mixture of Experts

Phi-3.5 MoE 42B

Phi-3.5 MoE — runs at 7B speed but with 42B total capacity. Best of both worlds.

chatcodingreasoningThinking

41.9B

Parameters (6.6B active)

128K

Context length

8

Benchmarks

14

Quantizations

200K

HF downloads

Architecture

MoE

Released

2024-08-20

Layers

32

KV Heads

8

Head Dim

128

Family

phi

Quantization Options

Quant	Bits	VRAM	Quality
IQ3_XXS	3.25	17.5 GB	low
IQ3_XS	3.5	18.8 GB	low
Q3_K_S	3.64	19.6 GB	low
IQ3_M	3.76	20.2 GB	low
Q3_K_M	4	21.4 GB	low
Q3_K_L	4.3	23.0 GB	moderate
IQ4_XS	4.46	23.8 GB	moderate
Q4_K_S	4.67	24.9 GB	moderate
Q4_K_M	4.89	26.1 GB	good
Q5_K_S	5.57	29.7 GB	good
Q5_K_M	5.7	30.3 GB	good
Q6_K	6.56	34.8 GB	excellent
Q8_0	8.5	45.0 GB	lossless
FP16	16	84.3 GB	lossless

Select your GPU above to see speed estimates and compatibility for each quantization.

▸ READY TO RUN THIS?RENT BY THE HOUR

RENT A GPU AND RUN PHI-3.5 MOE 42B NOW

Rent on RunPod →Or Vast.ai →

Spin up an A100 / H100 / 4090 in ~60s. Pay by the second. Cancel anytime.

Community Ratings

Loading ratings...

Benchmarks (8)

IFEval79.0

HumanEval77.0

BBH72.0

MATH66.0

MMLU-PRO60.8

GPQA43.0

BigCodeBench38.2

MUSR17.3

Run this model

▸Easiest way to get started·Beginners

curl -fsSL https://ollama.com/install.sh | sh

$ollama run phi:42b-q4_K_M

Tag may need adjustment — check ollama.com/library/phi for available tags.

▸ SETUP GUIDE

>_

Auto-setup with fitmyllm CLI

Detects your GPU, recommends the best model, downloads it, and starts chatting — zero config. Benchmarks your speed and contributes anonymous data to improve predictions.

pip install fitmyllmthen run fitmyllmLearn more

Auto-detect GPULive tok/s in chatSpeed benchmarks9 inference engines

HuggingFace GGUF Downloads Build Hardware

GPUs that can run this model

At Q4_K_M quantization. Sorted by minimum VRAM.

AMD Radeon PRO V710

28 GB VRAM • 504 GB/s

AMD

Amazon Newegg eBay PCPartPicker

NVIDIA RTX 5090

32 GB VRAM • 1792 GB/s

NVIDIA

$1999

Amazon Newegg eBay PCPartPicker

Apple M1 Max (32GB)

32 GB VRAM • 400 GB/s

APPLE

$1499

Apple M2 Max (32GB)

32 GB VRAM • 400 GB/s

APPLE

$1799

NVIDIA V100 SXM2 32GB

32 GB VRAM • 900 GB/s

NVIDIA

$3500

Apple M2 Pro (32GB)

32 GB VRAM • 200 GB/s

APPLE

$1499

Apple M4 (32GB)

32 GB VRAM • 120 GB/s

APPLE

$1199

NVIDIA Tesla V100 DGXS 32 GB

32 GB VRAM • 897 GB/s

NVIDIA

NVIDIA Tesla V100 PCIe 32 GB

32 GB VRAM • 897 GB/s

NVIDIA

NVIDIA Tesla V100 SXM2 32 GB

32 GB VRAM • 898 GB/s

NVIDIA

NVIDIA Tesla V100 SXM3 32 GB

32 GB VRAM • 981 GB/s

NVIDIA

AMD Radeon Instinct MI60

32 GB VRAM • 1020 GB/s

AMD

NVIDIA Tesla V100S PCIe 32 GB

32 GB VRAM • 1130 GB/s

NVIDIA

AMD Radeon Instinct MI100

32 GB VRAM • 1230 GB/s

AMD

$5000

NVIDIA RTX 5000 Ada Generation

32 GB VRAM • 576 GB/s

NVIDIA

$4000

Amazon Newegg eBay PCPartPicker

NVIDIA GeForce RTX 5090

32 GB VRAM • 1790 GB/s

NVIDIA

$1999

Amazon Newegg eBay PCPartPicker

NVIDIA GeForce RTX 5090 D

32 GB VRAM • 1790 GB/s

NVIDIA

$1999

Amazon Newegg eBay PCPartPicker

NVIDIA Jetson AGX Xavier 32 GB

32 GB VRAM • 136 GB/s

NVIDIA

Amazon Newegg eBay PCPartPicker

NVIDIA Quadro GV100

32 GB VRAM • 868 GB/s

NVIDIA

Amazon Newegg eBay PCPartPicker

NVIDIA TITAN V CEO Edition

32 GB VRAM • 868 GB/s

NVIDIA

Amazon Newegg eBay PCPartPicker

NVIDIA Tesla PG500-216

32 GB VRAM • 1130 GB/s

NVIDIA

NVIDIA Tesla PG503-216

32 GB VRAM • 1130 GB/s

NVIDIA

AMD Radeon Pro Vega II

32 GB VRAM • 825 GB/s

AMD

Amazon Newegg eBay PCPartPicker

AMD Radeon Pro Vega II Duo

32 GB VRAM • 1020 GB/s

AMD

Amazon Newegg eBay PCPartPicker

AMD Radeon PRO V620

32 GB VRAM • 512 GB/s

AMD

Amazon Newegg eBay PCPartPicker

AMD Radeon PRO W6800

32 GB VRAM • 512 GB/s

AMD

$2249

Amazon Newegg eBay PCPartPicker

AMD Radeon Pro W6800X

32 GB VRAM • 512 GB/s

AMD

Amazon Newegg eBay PCPartPicker

AMD Radeon Pro W6800X Duo

32 GB VRAM • 512 GB/s

AMD

Amazon Newegg eBay PCPartPicker

AMD Radeon Pro W6900X

32 GB VRAM • 512 GB/s

AMD

Amazon Newegg eBay PCPartPicker

NVIDIA Jetson AGX Orin 32 GB

32 GB VRAM • 205 GB/s

NVIDIA

Amazon Newegg eBay PCPartPicker

Find the best GPU for Phi-3.5 MoE 42B

Build Hardware for Phi-3.5 MoE 42B

Model Card

View on HuggingFace

Phi-3.5 MoE — runs at 7B speed but with 42B total capacity. Best of both worlds.

▸ SPEC SHEET

Phi-3.5 MoE 42B — 41.9B MoE.

▸ SPECIFICATIONS

PARAMETERS: 41.9B (6.6B active)
ARCHITECTURE: Mixture of Experts
CONTEXT LENGTH: 128K tokens
CAPABILITIES: chat, coding, reasoning
RELEASE DATE: 2024-08-20
PROVIDER: Microsoft
FAMILY: phi

▸ VRAM REQUIREMENTS

QUANT	BPW	VRAM	QUALITY
IQ3_XXS	3.25	17.5 GB	82%
IQ3_XS	3.5	18.8 GB	84%
Q3_K_S	3.64	19.6 GB	85%
IQ3_M	3.76	20.2 GB	86%
Q3_K_M	4	21.4 GB	88%
Q3_K_L	4.3	23.0 GB	90%
IQ4_XS	4.46	23.8 GB	92%
Q4_K_S	4.67	24.9 GB	93%
Q4_K_M	4.89	26.1 GB	94%
Q5_K_S	5.57	29.7 GB	96%
Q5_K_M	5.7	30.3 GB	96%
Q6_K	6.56	34.8 GB	97%
Q8_0	8.5	45.0 GB	100%
FP16	16	84.3 GB	100%

§ 01BENCHMARK SCORES

HumanEval77.0

MMLU-PRO60.8

MATH66.0

IFEval79.0

BBH72.0

GPQA43.0

MUSR17.3

BigCodeBench38.2

§ 03COMPATIBLE GPUs

30 @ Q4_K_M

AMD Radeon PRO V710

28 GB · 504 GB/s

NVIDIA RTX 5090

32 GB · 1792 GB/s

Apple M1 Max (32GB)

32 GB · 400 GB/s

Apple M2 Max (32GB)

32 GB · 400 GB/s

NVIDIA V100 SXM2 32GB

32 GB · 900 GB/s

Apple M2 Pro (32GB)

32 GB · 200 GB/s

Apple M4 (32GB)

32 GB · 120 GB/s

NVIDIA Tesla V100 DGXS 32 GB

32 GB · 897 GB/s

NVIDIA Tesla V100 PCIe 32 GB

32 GB · 897 GB/s

NVIDIA Tesla V100 SXM2 32 GB

32 GB · 898 GB/s

NVIDIA Tesla V100 SXM3 32 GB

32 GB · 981 GB/s

AMD Radeon Instinct MI60

32 GB · 1020 GB/s

NVIDIA Tesla V100S PCIe 32 GB

32 GB · 1130 GB/s

AMD Radeon Instinct MI100

32 GB · 1230 GB/s

NVIDIA RTX 5000 Ada Generation

32 GB · 576 GB/s