If eight training trials are waiting for one accelerator, finishing the first trial as quickly as possible may be the wrong immediate goal. A researcher deciding which trials deserve more compute needs some evidence from all eight. The scheduling metric should reflect that decision.
Gandiva emphasizes feedback-driven exploration and the repeated mini-batch structure of training. Its mechanisms use predictable iterations to support time-slicing and migration, allowing trials to expose early progress without first completing every earlier trial. The paper also distinguishes suspend/resume from simultaneously packing jobs that may interfere. Gandiva paper
Gandiva's paper reports a 26% aggregate utilization improvement in its 180-GPU evaluation. That result combines a cluster scheduler and implementation techniques; it is not the effect of round-robin order alone. Its early-feedback motivation is nevertheless directly testable without reproducing those GPU experiments, provided feedback is defined before comparing schedules.
Give feedback a precise definition
For a small synthetic calculation, suppose eight jobs arrive together. Each needs four one-second iterations. Define first feedback as completion of a job's first iteration, and assume that a switch between jobs costs 0.1 seconds. There is no initial setup charge.
Under run-to-completion, job 0 produces feedback at second 1 and finishes at second 4. Job 1 starts after the first switch, so its first feedback arrives at 5.1. Continue this pattern and the final job first reports at 29.7. All work finishes at 32.7: 32 seconds of useful iteration work plus seven switches.
Round-robin with a one-iteration quantum changes the story. First feedback arrives at 1, 2.1, 3.2, and so on, with the eighth result at 8.7. However, finishing all jobs now requires 31 switches. The makespan becomes 35.1 seconds.
The scheduler has bought much earlier coverage of the search space at the cost of 2.4 additional seconds of total elapsed time. No useful computation became faster. The order of useful computation changed.
The quantum is part of the experiment
A two-iteration quantum produces an intermediate point: the last first-feedback result arrives at 15.7 seconds, and completion takes 33.5 seconds. Comparing all three settings prevents the false conclusion that the smallest possible quantum is always best.
The included simulator measures these quantities directly from its event order. It switches only at iteration boundaries, so there is no fictional mid-kernel preemption. Its constant iteration duration and constant switch cost are modeling choices, not measured GPU behavior or claims about Gandiva's implementation.
First-iteration feedback is also a deliberately weak signal. A real search may need dozens of iterations, a validation pass, or a minimum training budget before a trial ranking is meaningful. Scheduling everything briefly is not helpful if those observations cannot support a decision. Change the feedback threshold before interpreting the latency chart.
There is another tradeoff beyond makespan: if early evidence allows poor trials to be stopped, time-slicing can reduce work that would otherwise have been wasted. That benefit requires a justified stopping policy and evidence about ranking reliability. This simulation stops no trials, keeping scheduling effects separate from search-quality claims.
Requiring a useful amount of training changes the answer
The extended event simulator tracks the completion of every iteration, rather than treating the first iteration as sufficient evidence by definition. It sweeps feedback thresholds of one, three, and four iterations; quanta of one through four; and switch costs of zero, 0.1, 0.5, and one second. That yields 48 fully specified schedules, all with the same 32 useful iteration-seconds.
At the original 0.1-second switch cost, the quantum minimizing the last-feedback time is one when feedback needs one iteration, three when it needs three, and four when it needs four. The corresponding times are 8.7, 24.7, and 32.7 seconds. A policy that was best for an early rough signal is not best for a decision that requires a more mature trial.
There is a simple reason the four-iteration case collapses to run-to-completion. Every job needs all its work before it can provide the requested signal, so the last-feedback objective is now the makespan objective. Splitting the work into additional visits adds switches without bringing that final signal forward. With zero switch cost, several schedules can tie; the code's reported first optimum is not evidence that the other tied choices are worse.
This is also why the simulator keeps feedback time and completion time in separate fields. A useful tuning dashboard should show both and explain what decision each feedback event permits. The right quantum depends on the reliability of that decision, not simply on how quickly a progress bar can move.