Validated Models

Model Name	Model Support	Continuous Batching Support
GPT2	✔️	✔️
Llama-3-8b	✔️	✔️
Llama-3-70b	✔️	✔️
Llama-2-70b	✔️	✔️
Llama-2-7b-chat-hf	✔️	✔️
Llama-2-13b-chat-hf	✔️	✔️
CodeLlama-7b-hf	✔️	✔️
CodeLlama-13b-hf	✔️	✔️
CodeLlama-34b-hf	✔️	✔️
Salesforce/codegen25-7b-mono_P	✔️	✔️
Salesforce/xgen-7b-8k-base	✔️	✔️
MPT-7b	✔️	✔️
Mistral-7B-Instruct-v0.1	✔️	✔️
Mixtral-8x7B	✔️	✔️
Vicuna-v0	✔️	✔️
Vicuna-v1.3	✔️	✔️
Vicuna-v1.5	✔️	✔️
Qwen2-1.5B-Instruct	✔️	✔️
StarCoder2-15B	✔️	✔️
Phi3-Mini-4K-Instruct	✔️	✔️
Codestral-22B-v0.1	✔️	✔️
Falcon-40b	✔️	✔️
GPT-J-6B	✔️	✔️
Jais-adapted-70b	✔️	✔️
Jais-adapted-13b-chat	✔️	✔️
Jais-adapted-7b	✔️	✔️

Models Coming Soon

• Chatglm2-6b

• Baichuan2-7B-Base