{
"nbformat": 4,
"nbformat_minor": 5,
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.14"
},
"gavel_analysis": {
"data_files": [
"gavel-fresh-trace.json",
"gavel-weight-summary.json",
"gavel-source.json"
],
"input_mount": "/work",
"execution": "saved-data analysis"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gavel: inspect the saved allocation trace\n",
"\n",
"This notebook recomputes the two **1:2:3** policy means in [How should AI jobs share different GPUs?](https://aryanputta.com/blog/gavel-trace), then lets you choose a timestep interval and a job. The inputs are the exact values saved in that article at commit `9df3f608a3e7b2e95312504f9d1cbf22e190091f`.\n",
"\n",
"Each score is effective throughput divided by that same job's expected rate at one-third time on each of K80, P100, and V100. The saved run used published policy code and measured throughput tables for an allocation-only numerical simulation. This notebook analyzes its outputs; it does not run GPU training or the round-based scheduler.\n",
"\n",
"The source and saved outputs are visible before execution. Run all cells to check the numbers, or change the interval and job in the query cell. The article's existing charts remain directly explorable without running Python.\n"
],
"id": "ec6cafa815b7"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Read the three inputs\n",
"\n",
"The site's runner mounts these files under `/work`: `gavel-fresh-trace.json`, `gavel-weight-summary.json`, and `gavel-source.json`. Locally, place them beside this notebook and run from that directory. The analysis uses the Python standard library; `IPython.display` renders the SVG outputs. There are no package installs or network calls in the notebook.\n"
],
"id": "d4fe435b39f2"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from pathlib import Path\n",
"from collections import defaultdict\n",
"from statistics import fmean\n",
"from math import isclose, isfinite, ceil\n",
"from html import escape\n",
"import csv\n",
"import hashlib\n",
"import json\n",
"\n",
"POLICIES = (\"with_perf\", \"agnostic\")\n",
"LABELS = {\"with_perf\": \"Throughput-aware\", \"agnostic\": \"Performance-agnostic\"}\n",
"COLORS = {\"with_perf\": \"#7d292e\", \"agnostic\": \"#246a73\"}\n",
"INPUT_FILES = (\"gavel-fresh-trace.json\", \"gavel-weight-summary.json\", \"gavel-source.json\")\n",
"\n",
"from IPython.display import SVG, display\n",
"\n",
"def load_inputs(folder):\n",
" \"\"\"Read the three mounted inputs and check their recorded content hashes.\"\"\"\n",
" folder = Path(folder)\n",
" trace, weights, source = [json.loads((folder / name).read_text()) for name in INPUT_FILES]\n",
" for name, expected in source[\"data_sha256\"].items():\n",
" actual = hashlib.sha256((folder / name).read_bytes()).hexdigest()\n",
" assert actual == expected, f\"Input hash changed: {name}\"\n",
" return trace, weights, source\n",
"\n",
"DATA_DIR = Path('/work') if (Path('/work') / INPUT_FILES[0]).is_file() else Path.cwd()\n",
"trace, weights, source = load_inputs(DATA_DIR)\n",
"print('Source article commit:', source['commit'])\n",
"print('Verified input hashes:', len(source['data_sha256']))\n",
"print('Saved job values:', sum(len(v['points']) for v in trace['policies'].values()))\n"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Source article commit: 9df3f608a3e7b2e95312504f9d1cbf22e190091f\n",
"Verified input hashes: 2\n",
"Saved job values: 1368\n"
]
}
],
"id": "242916de1a94"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Define the aggregation and checks\n",
"\n",
"At each timestep, sum the scores of jobs that have arrived. The published bar is the arithmetic mean of those 72 totals, so every timestep receives equal weight. Averaging all 684 job records directly would answer a different question because the number of active jobs changes.\n",
"\n",
"The checks verify unique job/timestep pairs, all 18 arrivals, both policy traces, finite nonnegative scores, and the two published means to an absolute tolerance of `1e-10`.\n"
],
"id": "d6ce763dd99f"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"def timestep_totals(trace, policy):\n",
" \"\"\"Sum the normalized scores of jobs that have arrived at each timestep.\"\"\"\n",
" assert policy in POLICIES\n",
" by_step = defaultdict(list)\n",
" for p in trace[\"policies\"][policy][\"points\"]:\n",
" by_step[p[\"t\"]].append(p[\"fresh\"])\n",
" return {t: sum(by_step[t]) for t in range(1, 73)}\n",
"\n",
"\n",
"def interval_summary(trace, start=1, end=72, job_id=None):\n",
" \"\"\"Aggregate every selected step; job means include only steps after arrival.\"\"\"\n",
" if type(start) is not int or type(end) is not int or not 1 <= start <= end <= 72:\n",
" raise ValueError(\"Choose integer timesteps with 1 <= start <= end <= 72.\")\n",
" if job_id is not None and (type(job_id) is not int or not 0 <= job_id < 18):\n",
" raise ValueError(\"Choose a job ID from 0 to 17, or None for all jobs.\")\n",
" result = []\n",
" for policy in POLICIES:\n",
" if job_id is None:\n",
" values = [v for t, v in timestep_totals(trace, policy).items() if start <= t <= end]\n",
" else:\n",
" values = [p[\"fresh\"] for p in trace[\"policies\"][policy][\"points\"]\n",
" if start <= p[\"t\"] <= end and p[\"job\"] == job_id]\n",
" result.append({\"policy\": policy, \"steps_with_values\": len(values),\n",
" \"mean_score\": fmean(values) if values else None,\n",
" \"min_score\": min(values) if values else None,\n",
" \"max_score\": max(values) if values else None})\n",
" return result\n",
"\n",
"\n",
"def print_summary(rows):\n",
" print(f\"{'Policy':<23} {'Steps':>5} {'Mean':>12} {'Min':>12} {'Max':>12}\")\n",
" for r in rows:\n",
" nums = [\"not arrived\" if r[k] is None else f\"{r[k]:.6f}\"\n",
" for k in (\"mean_score\", \"min_score\", \"max_score\")]\n",
" print(f\"{LABELS[r['policy']]:<23} {r['steps_with_values']:>5} \"\n",
" + \" \".join(f\"{n:>12}\" for n in nums))\n",
"\n",
"\n",
"def check_trace(trace, weights):\n",
" \"\"\"Check the saved trace's dimensions, arrivals, and published baseline means.\"\"\"\n",
" assert set(trace[\"policies\"]) == set(POLICIES)\n",
" for policy in POLICIES:\n",
" points = trace[\"policies\"][policy][\"points\"]\n",
" assert len(points) == 684\n",
" seen = set()\n",
" for p in points:\n",
" assert type(p[\"t\"]) is int and 1 <= p[\"t\"] <= 72\n",
" assert type(p[\"job\"]) is int and 0 <= p[\"job\"] < 18\n",
" assert isinstance(p[\"fresh\"], (float, int)) and isfinite(p[\"fresh\"]) and p[\"fresh\"] >= 0\n",
" key = (p[\"t\"], p[\"job\"])\n",
" assert key not in seen\n",
" seen.add(key)\n",
" expected_keys = {(t, job) for job in range(18) for t in range(1 + 4 * job, 73)}\n",
" assert seen == expected_keys, \"Missing, duplicate, or pre-arrival job values\"\n",
" totals = timestep_totals(trace, policy)\n",
" saved = next(r[\"mean_score\"] for r in weights\n",
" if r[\"scenario\"] == \"baseline_1_2_3\" and r[\"policy\"] == policy)\n",
" assert isclose(fmean(totals.values()), saved, rel_tol=0, abs_tol=1e-10)\n",
" expected_means = {\"with_perf\": 7.750821275892416, \"agnostic\": 6.441239770549949}\n",
" for policy, expected in expected_means.items():\n",
" assert isclose(fmean(timestep_totals(trace, policy).values()), expected,\n",
" rel_tol=0, abs_tol=1e-10)\n",
" assert len(weights) == 4\n",
" assert {(r[\"weights\"], r[\"policy\"]) for r in weights} == {\n",
" (w, p) for w in (\"1:2:3\", \"1:1:1\") for p in POLICIES}\n",
" return True\n",
"\n",
"assert check_trace(trace, weights)\n",
"assert interval_summary(trace, 1, 48, 12)[0]['mean_score'] is None\n",
"assert interval_summary(trace, 49, 72, 12)[0]['steps_with_values'] == 24\n",
"print('All numerical and trace-coverage assertions passed.')\n"
],
"execution_count": 2,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"All numerical and trace-coverage assertions passed.\n"
]
}
],
"id": "f684004f3efe"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Recompute the published baseline\n",
"\n",
"These are normalized progress scores, not raw tokens per second or GPU utilization. The comparison below describes one saved job sequence and one set of weights.\n"
],
"id": "3057a0298def"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"baseline = interval_summary(trace)\n",
"print('Weights 1:2:3; all 72 timesteps')\n",
"print_summary(baseline)\n",
"means = {row['policy']: row['mean_score'] for row in baseline}\n",
"aware_gain = 100 * (means['with_perf'] / means['agnostic'] - 1)\n",
"agnostic_drop = 100 * (1 - means['agnostic'] / means['with_perf'])\n",
"print(f'\\nThroughput-aware mean is {aware_gain:.3f}% above the agnostic mean.')\n",
"print(f'Agnostic mean is {agnostic_drop:.3f}% below the throughput-aware mean.')\n"
],
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Weights 1:2:3; all 72 timesteps\n",
"Policy Steps Mean Min Max\n",
"Throughput-aware 72 7.750821 1.719601 10.459325\n",
"Performance-agnostic 72 6.441240 1.000000 9.000000\n",
"\n",
"Throughput-aware mean is 20.331% above the agnostic mean.\n",
"Agnostic mean is 16.896% below the throughput-aware mean.\n"
]
}
],
"id": "3f9de44ce09a"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Draw the checked values\n",
"\n",
"Both lines use the saved 1:2:3 trace. Each plotted total is computed from its active jobs; no points are interpolated to create missing records. SVG point titles retain the timestep and value.\n"
],
"id": "1a5e8fc0634e"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"def line_svg(series, title, subtitle, start=1, end=72, reference=None):\n",
" \"\"\"Create a self-contained SVG from checked (timestep, value) pairs.\"\"\"\n",
" width, height, left, right, top, bottom = 900, 430, 76, 28, 98, 74\n",
" plot_w, plot_h = width-left-right, height-top-bottom\n",
" all_values = [v for pairs in series.values() for _, v in pairs]\n",
" ymax = max(1.0, ceil(max(all_values, default=1.0) * 1.12 * 2) / 2)\n",
" x = lambda t: left + ((t-start) / (end-start) if end > start else 0.5) * plot_w\n",
" y = lambda v: top + (1-v/ymax) * plot_h\n",
" chart_id = \"gavel-\" + hashlib.sha256(\n",
" json.dumps([title, subtitle, series], sort_keys=True).encode()\n",
" ).hexdigest()[:12]\n",
" parts = [f''])\n",
" return \"\\n\".join(parts)\n",
"\n",
"\n",
"def aggregate_svg(trace, start=1, end=72):\n",
" interval_summary(trace, start, end) # Validate the requested interval.\n",
" series = {p: [(t, v) for t, v in timestep_totals(trace, p).items() if start <= t <= end]\n",
" for p in POLICIES}\n",
" return line_svg(series, \"Aggregate normalized progress\", \n",
" f\"Weights 1:2:3 · sum of active jobs' scores · steps {start}–{end}\", start, end)\n",
"\n",
"\n",
"def job_svg(trace, job_id, start=1, end=72):\n",
" interval_summary(trace, start, end, job_id)\n",
" series = {policy: [(p[\"t\"], p[\"fresh\"]) for p in trace[\"policies\"][policy][\"points\"]\n",
" if p[\"job\"] == job_id and start <= p[\"t\"] <= end] for policy in POLICIES}\n",
" return line_svg(series, f\"Job {job_id}: normalized effective throughput\",\n",
" f\"Weights 1:2:3 · score relative to equal-time reference · steps {start}–{end}\",\n",
" start, end, reference=1.0)\n",
"\n",
"display(SVG(aggregate_svg(trace)))"
],
"execution_count": 4,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": ""
},
"metadata": {}
}
],
"id": "ac3ecd55c98b"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Query a different interval or job\n",
"\n",
"Edit `START_STEP`, `END_STEP`, or `JOB_ID`, then run this cell. Job IDs run from 0 to 17. A job's mean includes only the selected steps after its arrival; a pre-arrival interval reports “not arrived” instead of treating an absent job as a zero-throughput job. The aggregate mean still includes every selected timestep.\n"
],
"id": "962e9a48d2b6"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"START_STEP = 49\n",
"END_STEP = 72\n",
"JOB_ID = 12\n",
"\n",
"print(f'All active jobs, steps {START_STEP}–{END_STEP}:')\n",
"print_summary(interval_summary(trace, START_STEP, END_STEP))\n",
"print(f'\\nJob {JOB_ID}, selected steps after arrival only:')\n",
"print_summary(interval_summary(trace, START_STEP, END_STEP, JOB_ID))\n",
"display(SVG(job_svg(trace, JOB_ID, START_STEP, END_STEP)))\n"
],
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"All active jobs, steps 49–72:\n",
"Policy Steps Mean Min Max\n",
"Throughput-aware 24 9.876224 9.096838 10.459325\n",
"Performance-agnostic 24 8.728238 7.856588 9.000000\n",
"\n",
"Job 12, selected steps after arrival only:\n",
"Policy Steps Mean Min Max\n",
"Throughput-aware 24 1.130766 0.581074 1.527814\n",
"Performance-agnostic 24 0.888889 0.500000 1.000000\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": "",
"image/svg+xml": ""
},
"metadata": {}
}
],
"id": "089f9c5b29f6"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Compare the two saved weight scenarios\n",
"\n",
"The article also saves four means for weights **1:2:3** and **1:1:1**. The 1:2:3 means were independently recomputed above. The article contains only summary values for 1:1:1, so this cell compares those saved means without claiming to rerun that trace or solve new weights.\n",
"\n",
"Changing weights changes the allocation objective. These differences describe this workload response; they do not establish a general policy or fairness ranking. Target-share distances and raw GPU allocations are not inputs to this notebook.\n"
],
"id": "8b95cf00beac"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"def weight_comparison(weights):\n",
" \"\"\"Compare only the two scenarios with published saved summary values.\"\"\"\n",
" result = []\n",
" for policy in POLICIES:\n",
" values = {r[\"weights\"]: r[\"mean_score\"] for r in weights if r[\"policy\"] == policy}\n",
" result.append({\"policy\": policy, \"baseline_mean\": values[\"1:2:3\"],\n",
" \"uniform_saved_mean\": values[\"1:1:1\"],\n",
" \"change_percent\": 100*(values[\"1:1:1\"]/values[\"1:2:3\"]-1)})\n",
" return result\n",
"\n",
"print(f\"{'Policy':<23} {'1:2:3 recomputed':>17} {'1:1:1 saved':>15} {'Change':>10}\")\n",
"for row in weight_comparison(weights):\n",
" assert isclose(row['baseline_mean'], means[row['policy']], rel_tol=0, abs_tol=1e-10)\n",
" print(f\"{LABELS[row['policy']]:<23} {row['baseline_mean']:17.9f} \"\n",
" f\"{row['uniform_saved_mean']:15.9f} {row['change_percent']:9.3f}%\")\n"
],
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Policy 1:2:3 recomputed 1:1:1 saved Change\n",
"Throughput-aware 7.750821276 7.921449236 2.201%\n",
"Performance-agnostic 6.441239771 6.601999472 2.496%\n"
]
}
],
"id": "33c337cdfb8d"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Source trail\n",
"\n",
"- **Public references:** [Live article](https://aryanputta.com/blog/gavel-trace), [Gavel paper, OSDI 2020](https://www.usenix.org/system/files/osdi20-narayanan_deepak.pdf), and [Gavel source code](https://github.com/stanford-futuredata/gavel).\n",
"- **Exact input source:** `script#gavel-data` in the saved `source/gavel-trace.html`. Its Git blob SHA-1 is `572f09dc5a6399e5f80254791f0acbc1744798ba`.\n",
"- **Trace coverage:** 684 active-job records per policy, 1,368 total; 72 timesteps; 18 jobs. The full 1:2:3 series supports interval and job selection. The 1:1:1 series is not present in the article's embedded data.\n",
"- **Preserved reproduction:** [Download the original source, allocation data, and diagnostics](https://aryanputta.com/blog/assets/gavel-reproducibility.zip). This small analysis package supplements that download.\n",
"- **Portable checks:** `analysis.py` regenerates the CSV tables and SVG figures with Python's standard library. `validate.py` checks the exact extraction, source hashes, numerical assertions, figures, and executed notebook. `manifest.json` records the inputs and artifact hashes.\n"
],
"id": "1a4004190887"
}
]
}