LM Benchmarks
Beta| Model | Size | Release | Updated | LBCode | Aider | HumEval | HumEval+ | ArenaCode | LBReason | MMLU Pro | MMLU | GPQA | MATH | LBMath | Arena | AA Score | In Cost | Out Cost | Cutoff | Context | Org | Type | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
  | Claude 3.5 Sonnet | Latest | Oct 22, 2024 | 67.1% | 84.2% | 93.7% | 85.9% | 1321 | 58.7% | 78.0% | 88.0% | 65.0% | 78.3% | 51.3% | 1282 | 80 | $3 | $15 | Apr 24 | 200K | Anthropic | Proprietary | |
| | Claude 3.5 Sonnet | Old | Jun 20, 2024 | 60.9% | 77.4% | 92.0% | 1295 | 58.7% | 76.1% | 88.7% | 59.4% | 71.1% | 53.3% | 1268 | $3 | $15 | Apr 24 | 200K | Anthropic | Proprietary | |||
 | Qwen 2.5 Coder | 32B | Latest | Sep 19, 2024 | 56.9% | 73.7% | 92.7% | 87.2% | 1264 | 47.3% | 79.0% | 41.0% | 72.0% | 46.0% | 1220 | 70 | $0.2 | $0.2 | Sep 24 | 128K | Alibaba | Open Source | |
| Qwen 2.5 | 72B | Latest | Sep 19, 2024 | 56.6% | 55.6% | 59.1% | 51.2% | 1283 | 46.0% | 58.1% | 86.1% | 45.9% | 62.1% | 52.4% | 1259 | 75 | $0.38 | $0.57 | Sep 24 | 128K | Alibaba | Open Source |
 | Gemini Exp 1114 | Old | Nov 14, 2024 | 52.4% | 60.9% | 1324 | 54.7% | 54.9% | 1343 | 32K | Proprietary | ||||||||||||
  | GPT 4o | Old | Aug 6, 2024 | 51.4% | 71.4% | 90.0% | 87.2% | 1274 | 54.7% | 74.7% | 89.0% | 51.0% | 78.0% | 48.2% | 1265 | 77 | $2.5 | $10 | Oct 23 | 128K | OpenAI | Proprietary | |
| | Claude 3.5 Haiku | Latest | Oct 22, 2024 | 51.4% | 75.2% | 88.1% | 29.3% | 65.0% | 81.0% | 41.6% | 73.0% | 35.5% | 69 | $1 | $5 | Apr 24 | 200K | Anthropic | Proprietary | ||||
| | o1 Preview | Latest | Sep 12, 2024 | 50.8% | 79.7% | 92.4% | 89.0% | 1354 | 68.0% | 91.0% | 73.3% | 85.0% | 62.9% | 1333 | 85 | $15 | $60 | Oct 23 | 128K | OpenAI | Proprietary | ||
| Gemini Exp 1121 | Latest | Nov 21, 2024 | 50.4% | 57.9% | 1341 | 45.3% | 62.7% | 1365 | 32K | Proprietary | ||||||||||||
| QwQ Preview | 32B | Latest | Nov 28, 2024 | 50.0% | 75.0% | 65.2% | Alibaba | Open Source | ||||||||||||||
| Gemini 1.5 Pro | Latest | Sep 24, 2024 | 48.8% | 66.9% | 84.1% | 79.3% | 1290 | 46.0% | 75.8% | 86.0% | 61.0% | 86.5% | 57.4% | 1301 | 80 | $1.25 | $2.5 | Nov 23 | 2M | Proprietary | ||
| | o1 mini | Latest | Sep 12, 2024 | 48.0% | 70.7% | 92.4% | 89.0% | 1364 | 77.3% | 85.0% | 60.0% | 90.0% | 59.2% | 1308 | 82 | $3 | $12 | Oct 23 | 128K | OpenAI | Proprietary | ||
 | Mistral Large 2 | 123B | Old | Jul 24, 2024 | 47.1% | 60.2% | 91.3% | 62.2% | 1269 | 42.0% | 85.0% | 48.0% | 72.0% | 43.7% | 1250 | 74 | $2 | $6 | Jul 24 | 128K | Mistral | Proprietary | |
| | GPT 4o | 1.8T | Latest | Nov 20, 2024 | 46.1% | 72.9% | 90.0% | 1351 | 53.3% | 74.7% | 86.0% | 39.0% | 69.0% | 42.5% | 1361 | 71 | $2.5 | $10 | Oct 23 | 128K | OpenAI | Proprietary | |
 | DeepSeek 2.5 | 236B | Latest | Sep 12, 2024 | 45.5% | 72.2% | 89.0% | 83.5% | 1288 | 39.3% | 65.8% | 81.0% | 42.0% | 55.0% | 1221 | 66 | $0.14 | $0.28 | Nov 23 | 128K | DeepSeek | Proprietary | |
 | Llama 3.1 | 405B | Latest | Jul 23, 2024 | 43.8% | 66.2% | 89.0% | 1284 | 53.3% | 73.3% | 87.0% | 51.1% | 73.8% | 40.5% | 1266 | 72 | $5 | $15 | Dec 23 | 128K | Meta | Open Source | |
| | GPT 4o mini | 8B | Latest | Jul 18, 2024 | 43.1% | 55.6% | 87.2% | 83.5% | 1284 | 35.3% | 63.1% | 82.0% | 43.0% | 75.0% | 35.6% | 1272 | 71 | $0.15 | $0.6 | Oct 23 | 128K | OpenAI | Proprietary |
| Gemini 1.5 Flash | Latest | Sep 24, 2024 | 41.9% | 52.6% | 84.0% | 75.6% | 1254 | 50.0% | 67.3% | 81.0% | 50.0% | 79.0% | 47.2% | 1269 | 73 | $0.075 | $0.3 | Nov 23 | 1M | Proprietary | ||
| DeepSeek Coder 2 | 236B | Latest | Jul 22, 2024 | 41.5% | 72.9% | 90.2% | 82.3% | 1267 | 45.3% | 63.6% | 80.0% | 42.0% | 58.0% | 1213 | 67 | $0.14 | $0.28 | Nov 23 | 128K | DeepSeek | Proprietary | |
| Gemini 1.5 Pro | Old | Aug 27, 2024 | 40.9% | 66.9% | 1262 | 49.3% | 56.1% | 1260 | Nov 23 | 2M | Proprietary | |||||||||||
| Gemini 1.5 Flash | Old | Aug 27, 2024 | 40.6% | 52.6% | 1232 | 47.3% | 1227 | Nov 23 | 1M | Proprietary | ||||||||||||
| | Claude 3 Opus | Latest | Mar 4, 2024 | 38.6% | 68.4% | 84.9% | 77.4% | 1250 | 41.3% | 68.5% | 84.0% | 50.4% | 60.1% | 43.4% | 1248 | 70 | $15 | $75 | Aug 23 | 200K | Anthropic | Proprietary | |
| Mistral Large 3 | 123B | Latest | Nov 18, 2024 | 65.4% | 90.0% | 84.0% | 47.0% | 72.0% | 74 | $2 | $6 | 128K | Mistral | Proprietary | ||||||||
  | Grok-2 mini | Latest | Aug 13, 2024 | 37.5% | 54.9% | 85.7% | 1262 | 42.0% | 72.0% | 86.2% | 51.0% | 73.0% | 1266 | Aug 24 | 128K | xAI | Proprietary | ||||||
| | Grok 2 | Latest | Aug 13, 2024 | 36.2% | 58.6% | 88.4% | 80.5% | 1287 | 36.2% | 75.5% | 87.5% | 56.0% | 76.1% | 1290 | 70 | $5 | $15 | Aug 24 | 128K | xAI | Proprietary | ||
| Llama 3.1 | 70B | Latest | Jul 23, 2024 | 32.7% | 58.6% | 80.5% | 1244 | 40.7% | 62.8% | 84.0% | 43.0% | 60.0% | 34.4% | 1242 | 65 | $1 | $3 | Dec 23 | 128K | Meta | Open Source | |
| Gemini 1.5 Flash | 8B | Latest | Sep 24, 2024 | 28.7% | 38.3% | 1163 | 33.3% | 58.7% | 38.4% | 58.7% | 1177 | $0.0375 | $0.15 | Nov 23 | 1M | Proprietary | ||||||
| Mistral Nemo | 12B | Latest | Jul 24, 2024 | 28.7% | 33.1% | 25.3% | 68.0% | 33.0% | 40.0% | 52 | $0.15 | $0.15 | Apr 24 | 128K | Mistral | Proprietary | ||||||
| | Claude 3 Haiku | Old | Mar 7, 2024 | 24.5% | 47.4% | 75.9% | 1189 | 25.9% | 75.2% | 33.3% | 40.9% | 22.9% | 1179 | 54 | $0.25 | $1.25 | Aug 23 | 200K | Anthropic | Proprietary | |||
| Llama 3.1 | 8B | Latest | Jul 23, 2024 | 19.7% | 37.6% | 72.6% | 62.8% | 1174 | 15.3% | 71.0% | 27.0% | 64.0% | 16.6% | 1171 | 53 | $0.3 | $0.6 | Dec 23 | 128K | Meta | Open Source | |
| Pixtral Large | 123B | Latest | Nov 18, 2024 | 82.0% | 85.0% | 52.0% | 72.0% | 73 | $2 | $6 | 128K | Mistral | Proprietary | |||||||||
| Llama 3.2 | 3B | Latest | Sep 25, 2024 | 26.3% | 1048 | 63.4% | 21.0% | 49.0% | 1093 | 46 | $0.08 | $0.1 | Dec 23 | 128K | Meta | Open Source | ||||||
| Llama 3.2 | 1B | Latest | Sep 25, 2024 | 1009 | 49.3% | 14.0% | 27.0% | 1044 | 28 | $0.04 | $0.08 | Dec 23 | 128K | Meta | Open Source | |||||||
| Llama 3.2 (Vision) | 90B | Latest | Sep 25, 2024 | 86.0% | 43.0% | 68.0% | 67 | $0.9 | $0.9 | Dec 23 | 128K | Meta | Open Source | |||||||||
| Llama 3.2 (Vision) | 11B | Latest | Sep 25, 2024 | 73.0% | 26.0% | 51.9% | 54 | $0.19 | $0.19 | Dec 23 | 128K | Meta | Open Source | |||||||||
| R1 Lite Preview | Latest | Nov 20, 2024 | 58.5% | 91.6% | DeepSeek | Proprietary |
36 models