nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16
| benchmark | category | score | context | conf | evidence |
|---|---|---|---|---|---|
| ARC-Challenge | knowledge | 96.08 | ARC-Challenge | medium | | ARC-Challenge | 25-shot, acc_norm | **96.08** | 94.80 | 93.90 | |
| RULER | long-context | 93.17 | RULER 64K | medium | | RULER 64K | 0-shot | **93.17** | — | 81.58 | |
| GSM8K | math | 90.67 | GSM8K | medium | | GSM8K | 8-shot, EM | 90.67 | **90.75** | 82.60 | |
| RULER | long-context | 89.0 | RULER 128K | medium | | RULER 128K | 0-shot | **89.00** | 57.56 | 63.62 | |
| HellaSwag | knowledge | 88.97 | HellaSwag | medium | | HellaSwag | 10-shot, acc_norm | **88.97** | 84.69 | 87.70 | |
| MGSM | math | 87.47 | MGSM | medium | | MGSM | 8-shot, avg | **87.47** | 82.73 | 80.33 | |
| RULER | long-context | 86.18 | RULER 256K | medium | | RULER 256K | 0-shot | **86.18** | — | — | |
| MMLU | knowledge | 86.01 | MMLU | medium | | MMLU | 5-shot, acc | **86.01** | 81.00 | 81.00 | |
| MMLU | knowledge | 85.72 | MMLU Global Lite | medium | | MMLU Global Lite | 5-shot, avg | **85.72** | 74.94 | 79.25 | |
| PIQA | knowledge | 85.47 | PIQA | medium | | PIQA | 0-shot, acc_norm | **85.47** | 84.00 | 84.22 | |
| MATH | math | 84.84 | MATH | medium | | MATH | 4-shot, EM | **84.84** | 63.80 | 50.36 | |
| RULER | long-context | 82.16 | RULER 512K | medium | | RULER 512K | 0-shot | **82.16** | — | — | |
| HumanEval | coding | 79.4 | HumanEval | medium | | HumanEval | 0-shot, pass@1 n=32 | **79.40** | 70.10 | 76.30 | |
| WinoGrande | knowledge | 78.93 | WinoGrande | medium | | WinoGrande | 5-shot, acc | 78.93 | 78.37 | **83.82** | |
| MBPP | coding | 78.38 | MBPP-Sanitized | medium | | MBPP-Sanitized | 3-shot, pass@1 n=32 | **78.38** | 77.30 | 77.50 | |
| AGIEval | reasoning | 77.92 | AGIEval-En | medium | | AGIEval-En | 3/5-shot, CoT EM | **77.92** | 61.70 | 62.40 | |
| MMLU-Pro | knowledge | 75.65 | MMLU-Pro | medium | | MMLU-Pro | 5-shot, CoT EM | **75.65** | 62.10 | 58.20 | |
| RULER | long-context | 74.39 | RULER 1M | medium | | RULER 1M | 0-shot | **74.39** | — | — | |
| MATH | math | 70.0 | MATH Level 5 | medium | | MATH Level 5 | 4-shot, EM | **70.00** | 39.80 | 26.30 | |
| RULER | long-context | 64.0 | medium | RULER 64 | |
| GPQA Diamond | reasoning | 60.0 | GPQA-Diamond | medium | | GPQA-Diamond | 5-shot, CoT EM | **60.00** | 36.00 | 23.20 | |
| AIME 2024 | math | 53.33 | AIME 2024 | medium | | AIME 2024 | pass@32 | **53.33** | 30.00 | 20.00 | |
| OpenBookQA | knowledge | 50.2 | OpenBookQA | medium | | OpenBookQA | 0-shot, acc_norm | **50.20** | 47.00 | 48.60 | |
none scraped.