▸ SPEC SHEET
Falcon 40B — 40B Dense.
▸ SPECIFICATIONS
- PARAMETERS
- 40B
- ARCHITECTURE
- Dense Transformer
- CONTEXT LENGTH
- 8K tokens
- CAPABILITIES
- chat
- RELEASE DATE
- 2023-05-25
- PROVIDER
- TII
- FAMILY
- falcon
▸ VRAM REQUIREMENTS
| QUANT | BPW | VRAM | QUALITY |
|---|---|---|---|
| IQ3_XXS | 3.25 | 16.7 GB | 82% |
| IQ3_XS | 3.5 | 18.0 GB | 84% |
| Q3_K_S | 3.64 | 18.7 GB | 85% |
| IQ3_M | 3.76 | 19.3 GB | 86% |
| Q3_K_M | 4 | 20.5 GB | 88% |
| Q3_K_L | 4.3 | 22.0 GB | 90% |
| IQ4_XS | 4.46 | 22.8 GB | 92% |
| Q4_K_S | 4.67 | 23.8 GB | 93% |
| Q4_K_M | 4.89 | 24.9 GB | 94% |
| Q5_K_S | 5.57 | 28.3 GB | 96% |
| Q5_K_M | 5.7 | 29.0 GB | 96% |
| Q6_K | 6.56 | 33.3 GB | 97% |
| Q8_0 | 8.5 | 43.0 GB | 100% |
| FP16 | 16 | 80.5 GB | 100% |
§ 01BENCHMARK SCORES
HumanEval24.4
MMLU-PRO33.0
MATH1.8
IFEval27.0
BBH52.0
GPQA3.1
MUSR5.2
§ 03COMPATIBLE GPUs
30 @ Q4_K_MAMD Radeon PRO V710
28 GB · 504 GB/s
NVIDIA RTX 5090
32 GB · 1792 GB/s
Apple M1 Max (32GB)
32 GB · 400 GB/s
Apple M2 Max (32GB)
32 GB · 400 GB/s
NVIDIA V100 SXM2 32GB
32 GB · 900 GB/s
Apple M2 Pro (32GB)
32 GB · 200 GB/s
Apple M4 (32GB)
32 GB · 120 GB/s
NVIDIA Tesla V100 DGXS 32 GB
32 GB · 897 GB/s
NVIDIA Tesla V100 PCIe 32 GB
32 GB · 897 GB/s
NVIDIA Tesla V100 SXM2 32 GB
32 GB · 898 GB/s
NVIDIA Tesla V100 SXM3 32 GB
32 GB · 981 GB/s
AMD Radeon Instinct MI60
32 GB · 1020 GB/s
NVIDIA Tesla V100S PCIe 32 GB
32 GB · 1130 GB/s
AMD Radeon Instinct MI100
32 GB · 1230 GB/s
NVIDIA RTX 5000 Ada Generation
32 GB · 576 GB/s