How should AI jobs share different GPUs?

I’m interested in how AI workloads share limited compute, and in whether learned policies can make those systems adapt. Gavel is a useful systems paper to study because it separates two decisions: what allocation a policy wants, and how a scheduler can realize it on different hardware.

The mismatch Gavel addresses

A GPU is not a fixed amount of useful work. In Gavel’s measurements, a ResNet-50 sees nearly a tenfold throughputHow quickly a system makes progress on work. difference between a V100 and a K80, while an A3C reinforcement-learning workload sees about a twofold difference. The ranking can also change when cost matters: the newest device is not automatically best per dollar. A scheduler that counts accelerator time without modeling job-specific throughput can therefore give a job the wrong kind of capacity.

How Gavel turns policy into allocation

Gavel represents measured performance as a throughput table: the rate each job achieves on each accelerator type. A decision variable records the fraction of time a job, or a supported job combination, receives on each type. Multiplying those fractions by the measured rates gives effective throughputThe rate at which a system completes or advances work., the expected rate of progress across a job’s allocation.

The policy is expressed as an optimization objective over those allocations, with constraints that prevent jobs from exceeding their time share or the cluster from oversubscribing workers. Gavel transforms familiar objectives into heterogeneity-aware versions rather than hard-coding one scheduling rule. The paper covers least-attained-service and fair sharing, FIFO, makespan, cost subject to a completion-time SLO, and hierarchical policies such as fairness across teams with a different rule inside each team. Earlier schedulers each covered a narrower slice: Gandiva uses sharing without an explicit configurable objective, Tiresias and Themis target particular fairness policies without modeling accelerator affinity in the objective, and AlloX targets average completion time for single-worker jobs. Gavel’s claim is a reusable optimization framework for heterogeneous schedulingAllocating work across devices with different performance characteristics. across clusters.

That distinction between objective and mechanism is the central design choice. The policy computes target fractions; a separate scheduler tries to realize them. Gavel uses preemptive rounds, prioritizes jobs that have fallen behind their target share, avoids scheduling conflicting job combinations in the same round, and packs workers to limit fragmentation. Checkpointing lets training jobs pause and resume as placements change. The paper’s PyTorch interface, GavelIterator, coordinates workers and round boundaries through a narrow application interface.

Performance data are also part of the system. Users can provide throughput measurements, or Gavel’s throughput estimator can profile jobs. For space-sharing pairs whose performance is not known, it combines profiling with matrix completion and matches a new job to a similar reference job to estimate missing rates. The optimizer can also represent co-location combinations and placement effects for distributed jobs, rather than treating each GPU as interchangeable.

What the paper evaluates

The authors test the system on a physical cluster with 8 V100s, 16 P100s, and 24 K80s, and in simulation with 36 GPUs of each type. They use static traces where jobs are submitted together and continuous traces where new jobs arrive over time, across seven training workloads including image classification, language, recommendation, image translation, and A3C. Simulation runs vary input load and use three seeds for each load point.

They compare the policy objectives the system is designed to optimize: job completion time and makespan, along with cost and deadline violations for cost-aware policies. They also measure scheduling overhead, whether round-based placement tracks the optimizer’s allocation, how policy cost scales with job count, and how well the estimator predicts co-located throughput. On the physical cluster, heterogeneity-aware policies improve average job completion time by 1.5× and makespan by 1.2× in the tested traces. In larger simulations, the paper reports up to 3.5× lower average completion time at high load for its continuous single-worker trace. These are the authors’ results under their workloads and baselines.

For hierarchical fairness, Gavel uses an iterative water-filling procedure: a linear program divides allocation across entities while honoring each entity’s internal rule, and a mixed-integer program identifies jobs that have become bottlenecks. Figure 12 is one small hierarchical-fairness workload inside that evaluation: three entities have weights 1, 2, and 3; six jobs arrive for each entity, one every four timesteps, on a cluster with three GPUs of each type. The paper reports that its heterogeneity-agnostic version has about 17% lower total effective throughput than the heterogeneity-aware policy for this trace. The figure also shows how per-entity and within-entity fairness shift as jobs join.

Fresh allocation trace

This line chart follows one job’s normalized effective-throughput score over 72 simulated timesteps in the fresh local allocation run. Choose a policy and job, then move the timestep slider. A score of 1 means the job matches its equal-time reference rate; before its arrival, no score is drawn.

The run uses published Gavel policy code and published throughput measurements for an allocation-only numerical simulation; it does not execute the round-based scheduler. A job’s equal-time reference rate is its expected rate with one-third of its time on each of the K80, P100, and V100 GPU types.

Weighted allocation comparison

The allocation-only experiment compares entity weights 1:2:3 with 1:1:1 while holding the seed, job arrivals, published throughput table, cluster, and policy fixed. The chart compares the mean sum of active jobs’ normalized speed scores over 72 timesteps; higher values indicate greater aggregate progress relative to the per-job reference.

In this run, changing the weights alters the allocation objective while the mean summed score moves modestly. This single job mix and seed describe one workload response; they do not establish a general policy ranking.

Where I want to take this

I’m interested in how these allocation choices carry over to inference, where requests arrive over time and response delay matters. A next experiment could add a request trace, measure throughput and tail latency, and then ask whether a learned policy adapts as the workload mix changes. Gavel provides an optimization-based comparison point for that question; it is not a reinforcement-learning scheduler.