mistralai/Pixtral-12B-2409
| benchmark | category | score | context | conf | evidence |
|---|---|---|---|---|---|
| DocVQA | multimodal | 90.7 | **DocVQA** *(ANLS)* | medium | | **DocVQA** *(ANLS)* | 90.7 | <ins>**94.5**</ins> | 90.5 | 84.9 | 74.4 | |
| MMLU | knowledge | 68.5 | **MMLU** *(5-shot)* | medium | | **MMLU** *(5-shot)* | <ins>**69.2**</ins> | 68.5 | 67.9 | 63.5 | 63.6 | |
| MathVista | multimodal | 54.4 | **Mathvista** *(CoT)* | medium | | **Mathvista** *(CoT)* | <ins>**58.0**</ins> | 54.4 | 36.1 | 36.4 | 39.3 | |
| IFEval | instruction-following | 50.1 | **Text IF-Eval** | medium | | **Text IF-Eval** | <ins>**61.3**</ins> | 50.1 | 51.4 | 50.9 | 47.4 | |
| MMMU | multimodal | 47.6 | **MMMU** *(CoT)* | medium | | **MMMU** *(CoT)* | <ins>**52.5**</ins> | 47.6 | 45.1 | 40.3 | 38.3 | |
| IFEval | instruction-following | 38.9 | **MM IF-Eval** | medium | | **MM IF-Eval** | <ins>**52.7**</ins> | 38.9 | 42.5 | 41.2 | 31.4 | |
| ChartQA | multimodal | 38.6 | **ChartQA** *(CoT)* | medium | | **ChartQA** *(CoT)* | <ins>**81.8**</ins> | 38.6 | 67.1 | 72.0 | 67.7 | |
| MATH | math | 27.8 | **Math** *(Pass@1)* | medium | | **Math** *(Pass@1)* | <ins>**48.1**</ins> | 27.8 | 38.6 | 29.2 | 28.4 | |
| MT-Bench | chat | 6.41 | **Text MT-Bench** | medium | | **Text MT-Bench** | <ins>**7.68**</ins> | 6.41 | 6.94 | 6.27 | 6.31 | |
| MT-Bench | chat | 5.43 | **MM MT-Bench** | medium | | **MM MT-Bench** | <ins>**6.05**</ins> | 5.43 | 4.12 | 3.70 | 4.46 | |
none scraped.