The Server/Offline ratio is workload-specific
MLPerf Inference v6.1: released-data analysis
Aryan Putta · Research profile
Run from the extracted reproducibility bundle. Only Python's standard library is used. No GPU, network, model weights, APIs, or installation is required.
This notebook recomputes submitted measurements; it does not benchmark hardware. The source snapshot is pinned to 4bb63cd28eb136be221bc801a9e38c66c8a7553f. Upstream data and source files use Apache-2.0; see sources/LICENSE-MLCommons.md and provenance.json.
Verified upstream files: 15 CSV/JSON rows cross-checked: 492
1. Define the comparison before reading the ratio
Keep closed, available, datacenter, non-inferred (inferred = 0) rows with zero errors, no reported power result, Tokens/s units, and the Offline or Server scenario. Match every metadata field except Scenario, Result, Accuracy (the achieved metric), and Location (the scenario's log directory). The target-bearing model label remains in the key. Duplicate scenarios are errors, not opportunities to choose the fastest result.
source_rows: 492 closed_rows: 476 eligible_rows: 305 complete_pairs: 152 unmatched_row_ids: ['264'] pairs_with_server_greater_than_offline: 11 rows_with_measured_power: 0 Matched fields: 27
2. Inspect the eight small source rows
The subset retains every original field and the original JSON row IDs. The first three pairs hold the NVIDIA DGX B300 metadata constant within each workload. The fourth is a deliberately separate counterexample. Do not compare absolute rates across these systems as a hardware ranking.
B300-SXM-270GBx8_TRT deepseek-r1 59,867.8 / 70,132.3 = 85.36% B300-SXM-270GBx8_TRT gpt-oss-120b 100,630.0 / 106,637.0 = 94.37% B300-SXM-270GBx8_TRT llama2-70b-99.9 106,949.0 / 109,277.0 = 97.87% GB200-NVL72_GB200-186GB_aarch64x72_TRT gpt-oss-120b 899,005.0 / 883,761.0 = 101.72% B300 spread (percentage points): 12.51
3. Check accuracy targets, latency limits, and source logs
The pinned checker constants are parsed as an abstract syntax tree; upstream Python is not executed. Each reported accuracy metric must clear its model's threshold. All eight highlighted logs must say VALID, identify the intended scenario, and agree with the summary rate within rounding. The Server logs' actual TTFT and TPOT settings are checked too.
Row 285 | deepseek-r1 | Offline | VALID
Thresholds: {'exact_match': 80.544618, 'TOKENS_PER_SAMPLE': 3497.6046600000004}
Reported: {'exact_match': 80.78851412944394, 'TOKENS_PER_SAMPLE': 3793.9359617137648}
Row 286 | deepseek-r1 | Server | VALID
Thresholds: {'exact_match': 80.544618, 'TOKENS_PER_SAMPLE': 3497.6046600000004}
Reported: {'exact_match': 81.17593436645396, 'TOKENS_PER_SAMPLE': 3818.7203737465816}
Server limits (ms): {'ttft': 2000.0, 'tpot': 80.0}
Row 287 | gpt-oss-120b | Offline | VALID
Thresholds: {'exact_match': 82.2987}
Reported: {'exact_match': 83.43}
Row 288 | gpt-oss-120b | Server | VALID
Thresholds: {'exact_match': 82.2987}
Reported: {'exact_match': 83.702}
Server limits (ms): {'ttft': 3000.0, 'tpot': 80.0}
Row 289 | llama2-70b-99.9 | Offline | VALID
Thresholds: {'ROUGE1': 44.3867688, 'ROUGE2': 22.0131648, 'ROUGEL': 28.5875838, 'TOKENS_PER_SAMPLE': 265.005}
Reported: {'ROUGE1': 44.7951, 'ROUGE2': 22.3759, 'ROUGEL': 29.1534, 'TOKENS_PER_SAMPLE': 272.8}
Row 290 | llama2-70b-99.9 | Server | VALID
Thresholds: {'ROUGE1': 44.3867688, 'ROUGE2': 22.0131648, 'ROUGEL': 28.5875838, 'TOKENS_PER_SAMPLE': 265.005}
Reported: {'ROUGE1': 44.8452, 'ROUGE2': 22.4177, 'ROUGEL': 29.2149, 'TOKENS_PER_SAMPLE': 272.7}
Server limits (ms): {'ttft': 2000.0, 'tpot': 200.0}
Row 300 | gpt-oss-120b | Offline | VALID
Thresholds: {'exact_match': 82.2987}
Reported: {'exact_match': 84.038}
Row 301 | gpt-oss-120b | Server | VALID
Thresholds: {'exact_match': 82.2987}
Reported: {'exact_match': 83.914}
Server limits (ms): {'ttft': 3000.0, 'tpot': 80.0}
4. Test what must never be paired
Change the division, availability, units, inferred flag, measured-power flag, scenario, target label, accelerator count, framework, precision, or power note. The modified pair must be rejected. A duplicated Server row must fail instead of overwriting an entry.
13 positive, rejection, and duplicate-detection cases passed. Recomputed results exactly match the saved JSON. SVG generation exactly matches the saved figure.
5. Regenerated figure
The rich output below is generated from the same matched-row results.
Interpretation boundary
The ratios are 85.36%, 94.37%, and 97.87% for the three B300 workloads, a 12.51-point span. The separate GB200 GPT-OSS pair is 101.72%. Eleven of the 152 eligible pairs exceed 100%, so that value is not a universal bound on the released ratios.
These are independently submitted scenario results, not an isolated causal treatment. The summary does not establish identical batching or parallelism settings across scenarios, maximum attainable throughput, repeated-run variance, production latency, cost, or energy efficiency. All 492 source rows have has_power = false and null design_power_watts. TGP notes are not measured wall-plug power.
The full article supplies exact source links, limitations, and the matching rationale. See article.html, provenance.json, and README.md.
Python runs locally in this page. Saved results and figures are visible before execution.