allenai/Llama-3.1-Tulu-3-405B-DPO
| benchmark | category | score | context | conf | evidence |
|---|---|---|---|---|---|
| BBH | reasoning | 88.0 | **BigBenchHard (0 shot, CoT)** | medium | | **BigBenchHard (0 shot, CoT)** | 88.0 | 88.8 | 88.6 | 87.1 | 87.7 | **89.5** | 83.3 | |
| HumanEval | coding | 86.2 | **HumanEval (pass@10)** | medium | | **HumanEval (pass@10)** | 86.2 | 83.9 | 83.9 | 86.3 | **93.1** | 75.4 | 71.7 | 91.0 | |
| MMLU | knowledge | 84.4 | **MMLU (5 shot, CoT)** | medium | | **MMLU (5 shot, CoT)** | 84.4 | 86.6 | 87.0 | **88.0** | 84.9 | 82.1 | 87.9 | |
| HumanEval | coding | 81.4 | **HumanEval+ (pass@10)** | medium | | **HumanEval+ (pass@10)** | 81.4 | 78.6 | 79.2 | 82.9 | **89.7** | 69.1 | 67.0 | 88.5 | |
| GSM8K | math | 76.2 | **GSM8K (8 shot, CoT)** | medium | | **GSM8K (8 shot, CoT)** | 76.2 | 84.3 | **87.6** | 83.4 | 83.8 | 61.2 | 79.7 | 80.0 | |
| IFEval | instruction-following | 72.8 | **IFEval (prompt loose)** | medium | | **IFEval (prompt loose)** | 72.8 | 81.1 | **82.4** | 80.6 | 74.7 | 38.8 | 69.9 | 56.4 | |
| BBH | reasoning | 67.9 | **BigBenchHard (3 shot, CoT)** | medium | | **BigBenchHard (3 shot, CoT)** | **67.9** | 65.8 | 66.0 | 62.8 | 21.7 | 0.9 | 2.5 | 56.2 | |
| MMLU | knowledge | 65.9 | **MMLU (0 shot, CoT)** | medium | | **MMLU (0 shot, CoT)** | 65.9 | 68.7 | 68.2 | 71.2 | **76.6** | 62.0 | 74.6 | 68.5 | |
| MATH | math | 63.4 | **MATH (4 shot, Flex)** | medium | | **MATH (4 shot, Flex)** | 63.4 | 59.9 | 67.3 | 66.6 | 58.4 | **72.5** | 68.8 | |
| DROP | reasoning | 61.3 | **DROP (3 shot)** | medium | | **DROP (3 shot)** | 61.3 | 62.5 | **62.6** | 61.5 | 54.4 | 49.4 | 58.8 | 56.2 | |
| PopQA | knowledge | 55.7 | **PopQA (3 shot)** | medium | | **PopQA (3 shot)** | **55.7** | 55.4 | 55.5 | 52.9 | 54.2 | 44.9 | 53.6 | |
| TruthfulQA | safety | 46.8 | **TruthfulQA (6 shot)** | medium | | **TruthfulQA (6 shot)** | 46.8 | 56.1 | 55.0 | 55.1 | **63.1** | 57.0 | 61.4 | 55.5 | |
| MATH | math | 31.5 | **MATH (4 shot CoT, Flex)** | medium | | **MATH (4 shot CoT, Flex)** | 31.5 | 42.0 | **43.7** | 42.5 | 14.8 | 5.1 | 29.8 | 40.0 | |
| PopQA | knowledge | 29.3 | **PopQA (15 shot)** | medium | | **PopQA (15 shot)** | **29.3** | 29.3 | 29.1 | 20.2 | 18.1 | 22.5 | 28.3 | 20.2 | |
| AlpacaEval | chat | 12.4 | **AlpacaEval 2 (LC % win)** | medium | | **AlpacaEval 2 (LC % win)** | 12.4 | 33.5 | 34.5 | 24.2 | 29.0 | **49.0** | 43.7 | 31.4 | |
none scraped.