nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16
| benchmark | category | score | context | conf | evidence |
|---|---|---|---|---|---|
| ARC-Challenge | knowledge | 97.35 | ARC-Challenge | medium | | ARC-Challenge | *25-shot, acc_norm* | **97.35** | 95.22 | 97.27 | 95.82 | 96.59 | |
| RULER | long-context | 95.3 | RULER 64K | medium | | RULER 64K | *0-shot* | **95.30** | 93.30 | 90.11 | 93.79 | 16.12 | |
| RULER | long-context | 92.49 | RULER 128K | medium | | RULER 128K | *0-shot* | **92.49** | 91.88 | 55.77 | 88.61 | 0.00 | |
| HellaSwag | knowledge | 90.51 | HellaSwag | medium | | HellaSwag | *10-shot, acc_norm* | 90.51 | 89.44 | 88.88 | **90.92** | 90.17 | |
| MMLU | knowledge | 90.13 | MMLU Global Lite | medium | | MMLU Global Lite | *5-shot, avg* | **90.13** | 85.59 | 87.34 | 85.63 | 85.81 | |
| MMLU | knowledge | 89.08 | MMLU | medium | | MMLU | *5-shot, acc* | **89.08** | 87.82 | 87.35 | 87.60 | 86.50 | |
| GSM8K | math | 88.1 | GSM8K | medium | | GSM8K | *8-shot, CoT EM* | 88.10 | 84.38 | **91.21** | 91.05 | 85.37 | |
| MGSM | math | 87.73 | MGSM | medium | | MGSM | *8-shot, native CoT avg* | **87.73** | 82.33 | 82.93 | 85.20 | 81.27 | |
| RULER | long-context | 86.22 | RULER 256K | medium | | RULER 256K | *0-shot* | **86.22** | -- | 35.50 | -- | -- | |
| MBPP | coding | 85.97 | MBPP-Sanitized | medium | | MBPP-Sanitized | *3-shot pass@1 n=32, EvalPlus sanitized* | **85.97** | 58.66 | 84.08 | 72.14 | 76.69 | |
| RULER | long-context | 84.54 | RULER 512K | medium | | RULER 512K | *0-shot* | **84.54** | -- | -- | -- | -- | |
| HumanEval | coding | 83.84 | HumanEval | medium | | HumanEval | *sampled pass@1 n=32, EvalPlus sanitized* | **83.84** | 61.85 | 66.71 | 78.20 | 78.16 | |
| PIQA | knowledge | 83.79 | PIQA | medium | | PIQA | *0-shot, acc_norm* | 83.79 | 85.09 | 84.82 | **85.47** | 85.09 | |
| MATH | math | 82.0 | MATH | medium | | MATH | *4-shot, EM* | **82.00** | 60.12 | 62.88 | 68.40 | 57.58 | |
| WinoGrande | knowledge | 79.32 | WinoGrande | medium | | WinoGrande | *5-shot, acc* | 79.32 | 83.43 | 82.08 | 84.21 | **85.24** | |
| MMLU-Pro | knowledge | 79.07 | MMLU-Pro | medium | | MMLU-Pro | *5-shot, CoT EM* | **79.07** | 63.26 | 67.42 | 69.15 | 65.78 | |
| AGIEval | reasoning | 78.73 | AGIEval-En | medium | | AGIEval-En | *3/5-shot, CoT EM* | **78.73** | 70.13 | 69.30 | 72.55 | 70.06 | |
| RULER | long-context | 76.83 | RULER 1M | medium | | RULER 1M | *0-shot* | **76.83** | -- | -- | -- | -- | |
| RULER | long-context | 64.0 | medium | RULER 64 | |
| GPQA | reasoning | 50.0 | GPQA | medium | | GPQA | *5-shot, CoT EM* | **50.00** | 31.82 | 34.85 | 43.43 | 34.85 | |
| OpenBookQA | knowledge | 48.6 | OpenBookQA | medium | | OpenBookQA | *0-shot, acc_norm* | 48.60 | 48.20 | **51.40** | 50.80 | 49.60 | |
none scraped.