Phi-3.5 MoE 42B

Model Summary

Phi-3.5-MoE is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available documents - with a focus on very high-quality, reasoning dense data. The model supports multilingual and comes with 128K context length (in tokens). The model underwent a rigorous enhancement process, incorporating supervised fine-tuning, proximal policy optimization, and direct preference optimization to ensure precise instruction adherence and robust safety measures.

🏡 Phi-3 Portal

📰 Phi-3 Microsoft Blog

📖 Phi-3 Technical Report

👩‍🍳 Phi-3 Cookbook

🖥️ Try It

MoE references: 📜Phi-3.5-MoE Blog | 😁GRIN MoE

Phi-3.5: [mini-instruct]; [MoE-instruct] ; [vision-instruct]

Intended Uses

Primary Use Cases

The model is intended for commercial and research use in multiple languages. The model provides uses for general purpose AI systems and applications which require:

1. Memory/compute constrained environments
2. Latency bound scenarios
3. Strong reasoning (especially code, math and logic)

Our model is designed to accelerate research on language and multimodal models, for use as a building block for generative AI powered features.

Use Case Considerations

Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models as they select use cases, and evaluate and mitigate for accuracy, safety, and fariness before using within a specific downstream use case, particularly for high risk scenarios. Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case.

Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.

Usage

Requirements

Phi-3.5-MoE-instruct is integrated in the official version of transformers starting from 4.46.0. The current transformers version can be verified with: pip list | grep transformers.

Examples of required packages:

flash_attn==2.5.8
torch==2.3.1
accelerate==0.31.0
transformers==4.46.0

Phi-3.5-MoE-instruct is also available in Azure AI Studio

Tokenizer

Phi-3.5-MoE-Instruct supports a vocabulary size of up to 32064 tokens. The tokenizer files already provide placeholder tokens that can be used for downstream fine-tuning, but they can also be extended up to the model's vocabulary size.

Input Formats

Given the nature of the training data, the Phi-3.5-MoE-instruct model is best suited for prompts using the chat format as follows:

<|system|>
You are a helpful assistant.<|end|>
<|user|>
How to explain Internet for a medieval knight?<|end|>
<|assistant|>

Loading the model locally

After obtaining the Phi-3.5-MoE-instruct model checkpoints, users can use this sample code for inference.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline 

torch.random.manual_seed(0) 

model = AutoModelForCausalLM.from_pretrained( 
    "microsoft/Phi-3.5-MoE-instruct",  
    device_map="cuda",  
    torch_dtype="auto",  
    trust_remote_code=False,  
) 

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3.5-MoE-instruct") 

messages = [ 
    {"role": "system", "content": "You are a helpful AI assistant."}, 
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}, 
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."}, 
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"}, 
] 

pipe = pipeline( 
    "text-generation", 
    model=model, 
    tokenizer=tokenizer, 
) 

generation_args = { 
    "max_new_tokens": 500, 
    "return_full_text": False, 
    "temperature": 0.0, 
    "do_sample": False, 
} 

output = pipe(messages, **generation_args) 
print(output[0]['generated_text'])

Benchmarks

To understand the capabilities, we compare Phi-3.5-MoE with a set of models over a variety of benchmarks using our internal benchmark platform. At the high-level overview of the model quality on representative benchmarks:

Category	Benchmark	Phi-3.5-MoE-instruct	Mistral-Nemo-12B-instruct-2407	Llama-3.1-8B-instruct	Gemma-2-9b-It	Gemini-1.5-Flash	GPT-4o-mini-2024-07-18 (Chat)
Popular aggregated benchmark	Arena Hard	37.9	39.4	25.7	42.0	55.2	75.0
	BigBench Hard CoT (0-shot)	79.1	60.2	63.4	63.5	66.7	80.4
	MMLU (5-shot)	78.9	67.2	68.1	71.3	78.7	77.2
	MMLU-Pro (0-shot, CoT)	54.3	40.7	44.0	50.1	57.2	62.8
Reasoning	ARC Challenge (10-shot)	91.0	84.8	83.1	89.8	92.8	93.5
	BoolQ (2-shot)	84.6	82.5	82.8	85.7	85.8	88.7
	GPQA (0-shot, CoT)	36.8	28.6	26.3	29.2	37.5	41.1
	HellaSwag (5-shot)	83.8	76.7	73.5	80.9	67.5	87.1
	OpenBookQA (10-shot)	89.6	84.4	84.8	89.6	89.0	90.0
	PIQA (5-shot)	88.6	83.5	81.2	83.7	87.5	88.7
	Social IQA (5-shot)	78.0	75.3	71.8	74.7	77.8	82.9
	TruthfulQA (MC2) (10-shot)	77.5	68.1	69.2	76.6	76.6	78.2
	WinoGrande (5-shot)	81.3	70.4	64.7	74.0	74.7	76.9
Multilingual	Multilingual MMLU (5-shot)	69.9	58.9	56.2	63.8	77.2	72.9
	MGSM (0-shot CoT)	58.7	63.3	56.7	75.1	75.8	81.7
Math	GSM8K (8-shot, CoT)	88.7	84.2	82.4	84.9	82.4	91.3
	MATH (0-shot, CoT)	59.5	31.2	47.6	50.9	38.0	70.2
Long context	Qasper	40.0	30.7	37.2	13.9	43.5	39.8
	SQuALITY	24.1	25.8	26.2	0.0	23.5	23.8
Code Generation	HumanEval (0-shot)	70.7	63.4	66.5	61.0	74.4	86.6
	MBPP (3-shot)	80.8	68.1	69.4	69.3	77.5	84.1
Average		69.2	61.3	61.0	63.3	68.5	74.9

We take a closer look at different categories across 80 public benchmark datasets at the table below:

Category	Phi-3.5-MoE-instruct	Mistral-Nemo-12B-instruct-2407	Llama-3.1-8B-instruct	Gemma-2-9b-It	Gemini-1.5-Flash	GPT-4o-mini-2024-07-18 (Chat)
Popular aggregated benchmark	62.6	51.9	50.3	56.7	64.5	73.9
Reasoning	78.7	72.2	70.5	75.4	77.7	80.0
Language understanding	71.8	67.0	62.9	72.8	66.6	76.8
Robustness	75.6	65.2	59.8	64.7	68.9	77.5
Long context	25.5	24.5	25.5	0.0	27.0	25.4
Math	74.1	57.7	65.0	67.9	60.2	80.8
Code generation	68.3	56.9	65.8	58.3	66.8	69.9
Multilingual	65.8	55.3	47.5	59.6	64.3	76.6

...