Gavel allocation trace analysis

Python starts when you choose Run.

Gavel: inspect the saved allocation trace

This notebook recomputes the two 1:2:3 policy means in How should AI jobs share different GPUs?, then lets you choose a timestep interval and a job. The inputs are the exact values saved in that article at commit 9df3f608a3e7b2e95312504f9d1cbf22e190091f.

Each score is effective throughput divided by that same job's expected rate at one-third time on each of K80, P100, and V100. The saved run used published policy code and measured throughput tables for an allocation-only numerical simulation. This notebook analyzes its outputs; it does not run GPU training or the round-based scheduler.

The source and saved outputs are visible before execution. Run all cells to check the numbers, or change the interval and job in the query cell. The article's existing charts remain directly explorable without running Python.

1. Read the three inputs

The site's runner mounts these files under /work: gavel-fresh-trace.json, gavel-weight-summary.json, and gavel-source.json. Locally, place them beside this notebook and run from that directory. The analysis uses the Python standard library; IPython.display renders the SVG outputs. There are no package installs or network calls in the notebook.

from pathlib import Path
from collections import defaultdict
from statistics import fmean
from math import isclose, isfinite, ceil
from html import escape
import csv
import hashlib
import json

POLICIES = ("with_perf", "agnostic")
LABELS = {"with_perf": "Throughput-aware", "agnostic": "Performance-agnostic"}
COLORS = {"with_perf": "#7d292e", "agnostic": "#246a73"}
INPUT_FILES = ("gavel-fresh-trace.json", "gavel-weight-summary.json", "gavel-source.json")

from IPython.display import SVG, display

def load_inputs(folder):
    """Read the three mounted inputs and check their recorded content hashes."""
    folder = Path(folder)
    trace, weights, source = [json.loads((folder / name).read_text()) for name in INPUT_FILES]
    for name, expected in source["data_sha256"].items():
        actual = hashlib.sha256((folder / name).read_bytes()).hexdigest()
        assert actual == expected, f"Input hash changed: {name}"
    return trace, weights, source

DATA_DIR = Path('/work') if (Path('/work') / INPUT_FILES[0]).is_file() else Path.cwd()
trace, weights, source = load_inputs(DATA_DIR)
print('Source article commit:', source['commit'])
print('Verified input hashes:', len(source['data_sha256']))
print('Saved job values:', sum(len(v['points']) for v in trace['policies'].values()))

Source article commit: 9df3f608a3e7b2e95312504f9d1cbf22e190091f
Verified input hashes: 2
Saved job values: 1368

2. Define the aggregation and checks

At each timestep, sum the scores of jobs that have arrived. The published bar is the arithmetic mean of those 72 totals, so every timestep receives equal weight. Averaging all 684 job records directly would answer a different question because the number of active jobs changes.

The checks verify unique job/timestep pairs, all 18 arrivals, both policy traces, finite nonnegative scores, and the two published means to an absolute tolerance of 1e-10.

def timestep_totals(trace, policy):
    """Sum the normalized scores of jobs that have arrived at each timestep."""
    assert policy in POLICIES
    by_step = defaultdict(list)
    for p in trace["policies"][policy]["points"]:
        by_step[p["t"]].append(p["fresh"])
    return {t: sum(by_step[t]) for t in range(1, 73)}

def interval_summary(trace, start=1, end=72, job_id=None):
    """Aggregate every selected step; job means include only steps after arrival."""
    if type(start) is not int or type(end) is not int or not 1 <= start <= end <= 72:
        raise ValueError("Choose integer timesteps with 1 <= start <= end <= 72.")
    if job_id is not None and (type(job_id) is not int or not 0 <= job_id < 18):
        raise ValueError("Choose a job ID from 0 to 17, or None for all jobs.")
    result = []
    for policy in POLICIES:
        if job_id is None:
            values = [v for t, v in timestep_totals(trace, policy).items() if start <= t <= end]
        else:
            values = [p["fresh"] for p in trace["policies"][policy]["points"]
                      if start <= p["t"] <= end and p["job"] == job_id]
        result.append({"policy": policy, "steps_with_values": len(values),
                       "mean_score": fmean(values) if values else None,
                       "min_score": min(values) if values else None,
                       "max_score": max(values) if values else None})
    return result

def print_summary(rows):
    print(f"{'Policy':<23} {'Steps':>5} {'Mean':>12} {'Min':>12} {'Max':>12}")
    for r in rows:
        nums = ["not arrived" if r[k] is None else f"{r[k]:.6f}"
                for k in ("mean_score", "min_score", "max_score")]
        print(f"{LABELS[r['policy']]:<23} {r['steps_with_values']:>5} "
              + " ".join(f"{n:>12}" for n in nums))

def check_trace(trace, weights):
    """Check the saved trace's dimensions, arrivals, and published baseline means."""
    assert set(trace["policies"]) == set(POLICIES)
    for policy in POLICIES:
        points = trace["policies"][policy]["points"]
        assert len(points) == 684
        seen = set()
        for p in points:
            assert type(p["t"]) is int and 1 <= p["t"] <= 72
            assert type(p["job"]) is int and 0 <= p["job"] < 18
            assert isinstance(p["fresh"], (float, int)) and isfinite(p["fresh"]) and p["fresh"] >= 0
            key = (p["t"], p["job"])
            assert key not in seen
            seen.add(key)
        expected_keys = {(t, job) for job in range(18) for t in range(1 + 4 * job, 73)}
        assert seen == expected_keys, "Missing, duplicate, or pre-arrival job values"
        totals = timestep_totals(trace, policy)
        saved = next(r["mean_score"] for r in weights
                     if r["scenario"] == "baseline_1_2_3" and r["policy"] == policy)
        assert isclose(fmean(totals.values()), saved, rel_tol=0, abs_tol=1e-10)
    expected_means = {"with_perf": 7.750821275892416, "agnostic": 6.441239770549949}
    for policy, expected in expected_means.items():
        assert isclose(fmean(timestep_totals(trace, policy).values()), expected,
                       rel_tol=0, abs_tol=1e-10)
    assert len(weights) == 4
    assert {(r["weights"], r["policy"]) for r in weights} == {
        (w, p) for w in ("1:2:3", "1:1:1") for p in POLICIES}
    return True

assert check_trace(trace, weights)
assert interval_summary(trace, 1, 48, 12)[0]['mean_score'] is None
assert interval_summary(trace, 49, 72, 12)[0]['steps_with_values'] == 24
print('All numerical and trace-coverage assertions passed.')

All numerical and trace-coverage assertions passed.

3. Recompute the published baseline

These are normalized progress scores, not raw tokens per second or GPU utilization. The comparison below describes one saved job sequence and one set of weights.

Weights 1:2:3; all 72 timesteps
Policy                  Steps         Mean          Min          Max
Throughput-aware           72     7.750821     1.719601    10.459325
Performance-agnostic       72     6.441240     1.000000     9.000000

Throughput-aware mean is 20.331% above the agnostic mean.
Agnostic mean is 16.896% below the throughput-aware mean.

4. Draw the checked values

Both lines use the saved 1:2:3 trace. Each plotted total is computed from its active jobs; no points are interpolated to create missing records. SVG point titles retain the timestep and value.

def line_svg(series, title, subtitle, start=1, end=72, reference=None):
    """Create a self-contained SVG from checked (timestep, value) pairs."""
    width, height, left, right, top, bottom = 900, 430, 76, 28, 98, 74
    plot_w, plot_h = width-left-right, height-top-bottom
    all_values = [v for pairs in series.values() for _, v in pairs]
    ymax = max(1.0, ceil(max(all_values, default=1.0) * 1.12 * 2) / 2)
    x = lambda t: left + ((t-start) / (end-start) if end > start else 0.5) * plot_w
    y = lambda v: top + (1-v/ymax) * plot_h
    chart_id = "gavel-" + hashlib.sha256(
        json.dumps([title, subtitle, series], sort_keys=True).encode()
    ).hexdigest()[:12]
    parts = [f'<svg xmlns="http://www.w3.org/2000/svg" width="900" height="430" '
             f'viewBox="0 0 900 430" role="img" aria-labelledby="{chart_id}-title" '
             f'aria-describedby="{chart_id}-description">',
             f'<title id="{chart_id}-title">{escape(title)}</title>',
             f'<desc id="{chart_id}-description">{escape(subtitle)}. '
             'Scores use each job\'s equal-time K80, P100, V100 reference. Missing jobs are not drawn.</desc>',
             '<rect width="900" height="430" fill="white"/>',
             '<g font-family="system-ui,Arial,sans-serif" font-size="12" fill="#42484b">',
             f'<text x="24" y="30" font-size="20" font-weight="600">{escape(title)}</text>',
             f'<text x="24" y="53" fill="#636d72">{escape(subtitle)}</text>']
    for i, policy in enumerate(series):
        lx = 24 + 230*i
        dash = ' stroke-dasharray="6 4"' if policy == "agnostic" else ""
        parts.append(f'<line x1="{lx}" y1="76" x2="{lx+24}" y2="76" '
                     f'stroke="{COLORS[policy]}" stroke-width="3"{dash}/>')
        parts.append(f'<text x="{lx+32}" y="80">{LABELS[policy]}</text>')
    for i in range(5):
        val = ymax*i/4
        parts.append(f'<line x1="{left}" y1="{y(val):.2f}" x2="{width-right}" '
                     f'y2="{y(val):.2f}" stroke="#e5e8e9"/>')
        parts.append(f'<text x="{left-12}" y="{y(val)+4:.2f}" text-anchor="end">{val:.2f}</text>')
    ticks = sorted({round(start + (end-start)*i/4) for i in range(5)})
    for t in ticks:
        parts.append(f'<text x="{x(t):.2f}" y="{height-bottom+22}" text-anchor="middle">{t}</text>')
    parts.append(f'<text x="{left+plot_w/2}" y="{height-20}" text-anchor="middle">Simulated timestep</text>')
    if reference is not None:
        parts.append(f'<line x1="{left}" y1="{y(reference):.2f}" x2="{width-right}" '
                     f'y2="{y(reference):.2f}" stroke="#72797d" stroke-dasharray="3 4"/>')
    for policy, pairs in series.items():
        coords = " ".join(f"{x(t):.3f},{y(v):.3f}" for t, v in pairs)
        dash = ' stroke-dasharray="6 4"' if policy == "agnostic" else ""
        parts.append(f'<polyline points="{coords}" fill="none" stroke="{COLORS[policy]}" '
                     f'stroke-width="2.8"{dash}/>')
        for t, v in pairs:
            parts.append(f'<circle cx="{x(t):.3f}" cy="{y(v):.3f}" r="2.3" fill="{COLORS[policy]}">'
                         f'<title>{LABELS[policy]}, step {t}: {v:.12g}</title></circle>')
    if not all_values:
        parts.append(f'<text x="450" y="230" text-anchor="middle">The selected job has not arrived in this interval.</text>')
    parts.extend(['</g>', '</svg>'])
    return "\n".join(parts)

def aggregate_svg(trace, start=1, end=72):
    interval_summary(trace, start, end)  # Validate the requested interval.
    series = {p: [(t, v) for t, v in timestep_totals(trace, p).items() if start <= t <= end]
              for p in POLICIES}
    return line_svg(series, "Aggregate normalized progress", 
                    f"Weights 1:2:3 · sum of active jobs' scores · steps {start}–{end}", start, end)

def job_svg(trace, job_id, start=1, end=72):
    interval_summary(trace, start, end, job_id)
    series = {policy: [(p["t"], p["fresh"]) for p in trace["policies"][policy]["points"]
                       if p["job"] == job_id and start <= p["t"] <= end] for policy in POLICIES}
    return line_svg(series, f"Job {job_id}: normalized effective throughput",
                    f"Weights 1:2:3 · score relative to equal-time reference · steps {start}–{end}",
                    start, end, reference=1.0)

display(SVG(aggregate_svg(trace)))

Saved-data aggregate trace analysis from the reviewed Gavel allocation run.

5. Query a different interval or job

Edit START_STEP, END_STEP, or JOB_ID, then run this cell. Job IDs run from 0 to 17. A job's mean includes only the selected steps after its arrival; a pre-arrival interval reports “not arrived” instead of treating an absent job as a zero-throughput job. The aggregate mean still includes every selected timestep.

All active jobs, steps 49–72:
Policy                  Steps         Mean          Min          Max
Throughput-aware           24     9.876224     9.096838    10.459325
Performance-agnostic       24     8.728238     7.856588     9.000000

Job 12, selected steps after arrival only:
Policy                  Steps         Mean          Min          Max
Throughput-aware           24     1.130766     0.581074     1.527814
Performance-agnostic       24     0.888889     0.500000     1.000000

Saved-data selected-job analysis from the reviewed Gavel allocation run.

6. Compare the two saved weight scenarios

The article also saves four means for weights 1:2:3 and 1:1:1. The 1:2:3 means were independently recomputed above. The article contains only summary values for 1:1:1, so this cell compares those saved means without claiming to rerun that trace or solve new weights.

Changing weights changes the allocation objective. These differences describe this workload response; they do not establish a general policy or fairness ranking. Target-share distances and raw GPU allocations are not inputs to this notebook.

Policy                   1:2:3 recomputed     1:1:1 saved     Change
Throughput-aware              7.750821276     7.921449236     2.201%
Performance-agnostic          6.441239771     6.601999472     2.496%

Source trail

Public references: Live article, Gavel paper, OSDI 2020, and Gavel source code.
Exact input source: script#gavel-data in the saved source/gavel-trace.html. Its Git blob SHA-1 is 572f09dc5a6399e5f80254791f0acbc1744798ba.
Trace coverage: 684 active-job records per policy, 1,368 total; 72 timesteps; 18 jobs. The full 1:2:3 series supports interval and job selection. The 1:1:1 series is not present in the article's embedded data.
Preserved reproduction: Download the original source, allocation data, and diagnostics. This small analysis package supplements that download.
Portable checks: analysis.py regenerates the CSV tables and SVG figures with Python's standard library. validate.py checks the exact extraction, source hashes, numerical assertions, figures, and executed notebook. manifest.json records the inputs and artifact hashes.

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook