{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# The Server/Offline ratio is workload-specific\n", "## MLPerf Inference v6.1: released-data analysis\n", "**Aryan Putta** · [Research profile](https://aryanputta.com/research-profile/)\n", "\n", "Run from the extracted reproducibility bundle. Only Python's standard library is used. No GPU, network, model weights, APIs, or installation is required.\n", "\n", "This notebook recomputes submitted measurements; it does not benchmark hardware. The source snapshot is pinned to `4bb63cd28eb136be221bc801a9e38c66c8a7553f`. Upstream data and source files use Apache-2.0; see `sources/LICENSE-MLCommons.md` and `provenance.json`.\n" ], "id": "cell-00" }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Verified upstream files: 15\n", "CSV/JSON rows cross-checked: 492\n" ] } ], "source": [ "import json\n", "from pathlib import Path\n", "from analyze import analyze, load_rows, match_pairs, check_provenance, self_tests, FOCUS_IDS\n", "from make_plot import make_svg\n", "root = Path.cwd()\n", "assert (root / 'data/summary.json').exists(), 'Open this notebook from the extracted bundle folder.'\n", "print('Verified upstream files:', check_provenance(root))\n", "rows = load_rows(root)\n", "print('CSV/JSON rows cross-checked:', len(rows))\n" ], "id": "cell-01" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Define the comparison before reading the ratio\n", "\n", "Keep closed, available, datacenter, non-inferred (`inferred = 0`) rows with zero errors, no reported power result, Tokens/s units, and the Offline or Server scenario. Match every metadata field except Scenario, Result, Accuracy (the achieved metric), and Location (the scenario's log directory). The target-bearing model label remains in the key. Duplicate scenarios are errors, not opportunities to choose the fastest result.\n" ], "id": "cell-02" }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "source_rows: 492\n", "closed_rows: 476\n", "eligible_rows: 305\n", "complete_pairs: 152\n", "unmatched_row_ids: ['264']\n", "pairs_with_server_greater_than_offline: 11\n", "rows_with_measured_power: 0\n", "Matched fields: 27\n" ] } ], "source": [ "results = analyze(root)\n", "for key in ['source_rows', 'closed_rows', 'eligible_rows', 'complete_pairs', 'unmatched_row_ids', 'pairs_with_server_greater_than_offline', 'rows_with_measured_power']:\n", " print(f'{key}: {results[key]}')\n", "print('Matched fields:', len(results['match_fields']))\n" ], "id": "cell-03" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Inspect the eight small source rows\n", "\n", "The subset retains every original field and the original JSON row IDs. The first three pairs hold the NVIDIA DGX B300 metadata constant within each workload. The fourth is a deliberately separate counterexample. Do not compare absolute rates across these systems as a hardware ranking.\n" ], "id": "cell-04" }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "B300-SXM-270GBx8_TRT deepseek-r1\n", " 59,867.8 / 70,132.3 = 85.36%\n", "B300-SXM-270GBx8_TRT gpt-oss-120b\n", " 100,630.0 / 106,637.0 = 94.37%\n", "B300-SXM-270GBx8_TRT llama2-70b-99.9\n", " 106,949.0 / 109,277.0 = 97.87%\n", "GB200-NVL72_GB200-186GB_aarch64x72_TRT gpt-oss-120b\n", " 899,005.0 / 883,761.0 = 101.72%\n", "B300 spread (percentage points): 12.51\n" ] } ], "source": [ "small = json.loads((root / 'data/selected_rows.json').read_text())\n", "assert list(small) == FOCUS_IDS\n", "assert all(small[i] == rows[i] for i in FOCUS_IDS)\n", "focused, unmatched = match_pairs(small)\n", "assert not unmatched and len(focused) == 4\n", "for p in focused:\n", " print(p['metadata']['Platform'], p['metadata']['MlperfModel'])\n", " print(f\" {p['server_tokens_per_second']:,.1f} / {p['offline_tokens_per_second']:,.1f} = {p['server_over_offline_percent']:.2f}%\")\n", "print('B300 spread (percentage points):', round(results['b300_ratio_span_percentage_points'], 2))\n" ], "id": "cell-05" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Check accuracy targets, latency limits, and source logs\n", "\n", "The pinned checker constants are parsed as an abstract syntax tree; upstream Python is not executed. Each reported accuracy metric must clear its model's threshold. All eight highlighted logs must say VALID, identify the intended scenario, and agree with the summary rate within rounding. The Server logs' actual TTFT and TPOT settings are checked too.\n" ], "id": "cell-06" }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Row 285 | deepseek-r1 | Offline | VALID\n", " Thresholds: {'exact_match': 80.544618, 'TOKENS_PER_SAMPLE': 3497.6046600000004}\n", " Reported: {'exact_match': 80.78851412944394, 'TOKENS_PER_SAMPLE': 3793.9359617137648}\n", "Row 286 | deepseek-r1 | Server | VALID\n", " Thresholds: {'exact_match': 80.544618, 'TOKENS_PER_SAMPLE': 3497.6046600000004}\n", " Reported: {'exact_match': 81.17593436645396, 'TOKENS_PER_SAMPLE': 3818.7203737465816}\n", " Server limits (ms): {'ttft': 2000.0, 'tpot': 80.0}\n", "Row 287 | gpt-oss-120b | Offline | VALID\n", " Thresholds: {'exact_match': 82.2987}\n", " Reported: {'exact_match': 83.43}\n", "Row 288 | gpt-oss-120b | Server | VALID\n", " Thresholds: {'exact_match': 82.2987}\n", " Reported: {'exact_match': 83.702}\n", " Server limits (ms): {'ttft': 3000.0, 'tpot': 80.0}\n", "Row 289 | llama2-70b-99.9 | Offline | VALID\n", " Thresholds: {'ROUGE1': 44.3867688, 'ROUGE2': 22.0131648, 'ROUGEL': 28.5875838, 'TOKENS_PER_SAMPLE': 265.005}\n", " Reported: {'ROUGE1': 44.7951, 'ROUGE2': 22.3759, 'ROUGEL': 29.1534, 'TOKENS_PER_SAMPLE': 272.8}\n", "Row 290 | llama2-70b-99.9 | Server | VALID\n", " Thresholds: {'ROUGE1': 44.3867688, 'ROUGE2': 22.0131648, 'ROUGEL': 28.5875838, 'TOKENS_PER_SAMPLE': 265.005}\n", " Reported: {'ROUGE1': 44.8452, 'ROUGE2': 22.4177, 'ROUGEL': 29.2149, 'TOKENS_PER_SAMPLE': 272.7}\n", " Server limits (ms): {'ttft': 2000.0, 'tpot': 200.0}\n", "Row 300 | gpt-oss-120b | Offline | VALID\n", " Thresholds: {'exact_match': 82.2987}\n", " Reported: {'exact_match': 84.038}\n", "Row 301 | gpt-oss-120b | Server | VALID\n", " Thresholds: {'exact_match': 82.2987}\n", " Reported: {'exact_match': 83.914}\n", " Server limits (ms): {'ttft': 3000.0, 'tpot': 80.0}\n" ] } ], "source": [ "for c in results['accuracy_and_log_checks']:\n", " print(f\"Row {c['row_id']} | {c['model']} | {c['scenario']} | {c['log_result']}\")\n", " print(' Thresholds:', c['thresholds'])\n", " print(' Reported:', c['reported_accuracy'])\n", " if 'limits_ns' in c:\n", " print(' Server limits (ms):', {k: v / 1_000_000 for k, v in c['limits_ns'].items()})\n" ], "id": "cell-07" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Test what must never be paired\n", "\n", "Change the division, availability, units, inferred flag, measured-power flag, scenario, target label, accelerator count, framework, precision, or power note. The modified pair must be rejected. A duplicated Server row must fail instead of overwriting an entry.\n" ], "id": "cell-08" }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "13 positive, rejection, and duplicate-detection cases passed.\n", "Recomputed results exactly match the saved JSON.\n", "SVG generation exactly matches the saved figure.\n" ] } ], "source": [ "assert self_tests(rows) == 13\n", "print('13 positive, rejection, and duplicate-detection cases passed.')\n", "assert results == json.loads((root / 'results.json').read_text())\n", "print('Recomputed results exactly match the saved JSON.')\n", "assert make_svg(results) == (root / 'scenario-ratios.svg').read_text()\n", "print('SVG generation exactly matches the saved figure.')\n" ], "id": "cell-09" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Regenerated figure\n", "\n", "The rich output below is generated from the same matched-row results.\n" ], "id": "cell-10" }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "output_type": "execute_result", "execution_count": 6, "metadata": {}, "data": { "image/svg+xml": "\nServer throughput relative to the matched Offline run\nThree NVIDIA DGX B300 submissions have Server to Offline ratios of 85.36, 94.37 and 97.87 percent. A separate NVIDIA GB200 GPT-OSS submission is 101.72 percent. Each percentage compares scenarios within one exact metadata group, never two systems.\n\n\nThe scenario ratio changes with the workload\nServer ÷ Offline × 100 · released MLPerf v6.1 results · higher is not a hardware ranking\nNVIDIA DGX B300 · 1 node · 8 × B300-SXM-270GB · FP4\n0%25%50%75%100%DeepSeek-R185.36%GPT-OSS-120B94.37%Llama 2 70B · 99.9 target97.87%GPT-OSS-120B101.72%\nSeparate counterexample: NVIDIA GB200 NVL72 · 18 nodes · 72 × GB200 · FP4\nEach bar retains its own model, accuracy target, platform, precision and software metadata.\nClosed division · available systems · inferred = 0 · Tokens/s · has_power = false\nSource: MLCommons inference_results_v6.1, commit 4bb63cd28eb13. Analysis and figure: Aryan Putta.\n", "text/plain": "Regenerated scenario-ratios.svg" } } ], "source": [ "class LocalSVG:\n", " def _repr_mimebundle_(self, include=None, exclude=None):\n", " return {'image/svg+xml': make_svg(results), 'text/plain': 'Regenerated scenario-ratios.svg'}\n", "LocalSVG()\n", "\n", "import base64\n", "from pathlib import Path\n", "print('__NOTEBOOK_IMAGE__svg+xml:' + base64.b64encode(Path('scenario-ratios.svg').read_bytes()).decode())\n" ], "id": "cell-11" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Interpretation boundary\n", "\n", "The ratios are 85.36%, 94.37%, and 97.87% for the three B300 workloads, a 12.51-point span. The separate GB200 GPT-OSS pair is 101.72%. Eleven of the 152 eligible pairs exceed 100%, so that value is not a universal bound on the released ratios.\n", "\n", "These are independently submitted scenario results, not an isolated causal treatment. The summary does not establish identical batching or parallelism settings across scenarios, maximum attainable throughput, repeated-run variance, production latency, cost, or energy efficiency. All 492 source rows have `has_power = false` and null `design_power_watts`. TGP notes are not measured wall-plug power.\n", "\n", "The full article supplies exact source links, limitations, and the matching rationale. See `article.html`, `provenance.json`, and `README.md`.\n" ], "id": "cell-12" } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12.14" }, "source_revision": "4bb63cd28eb136be221bc801a9e38c66c8a7553f", "execution": { "method": "Ordered Python cell execution with execute_notebook.py", "engine": "CPython", "version": "3.12.14", "code_cells_executed": 6, "network_used": false } }, "nbformat": 4, "nbformat_minor": 5 }