Research / Agent systems
The gap between a fresh screenshot and a safe tap
Re-observation narrows the stale-state window. Only an executor-enforced revision check closes it in this timing model.
A tap can be geometrically correct for the screenshot and wrong for the screen at execution time. If a list reorders during inference, the old target position may now belong to another item. Measuring only whether the coordinate matches the screenshot misses the temporal part of grounding.
Inspect a case
Put time into the state
AndroidWorld provides screenshot and accessibility-tree observations, gesture actions, and tasks with controlled setup and state-based rewardsChecking whether the resulting environment satisfies the requested outcome.. Its parameterized tasks motivate testing more than one fixed interface condition. The latency experiment here is a proposed, pure-Python extension of that idea, not a result measured in AndroidWorld or an Android emulator. AndroidWorld, Sections 3.2–3.4
The synthetic interface contains two slots. Initially the target occupies the left slot and a distractor occupies the right slot. At a scheduled time they swap once and remain in their new positions. An observation returns the target's slot and a revision number. The simulator knows the full schedule, but the scripted clicking policies receive only observations at their specified times.
Twenty schedules are drawn once from a fixed seed. Each chooses a swap time from 25, 75, 125, 275, or 525 milliseconds. The same schedules are reused for every policy and every delay, avoiding a comparison in which one policy accidentally receives easier timing cases.
Compare three contracts
All policies execute at delay + 30 milliseconds. The stale policy chooses from the initial observationWhether the evidence an agent uses still describes the world when it acts. at time zero. The refreshed policy observes at the chosen delay and then spends thirty milliseconds in a final action pipeline. The guarded policy uses the refreshed observation but asks the executor to reject the action if the revision has changed before execution.
Consider a swap at 125 milliseconds and a delay of 100 milliseconds. Execution occurs at 130. The stale policy points left and hits the distractor. The refreshed policy also points left because its observation at 100 precedes the swap. The guarded policy notices that revision zero became revision one and cancels. A recent observation reduced the vulnerable interval but did not eliminate it.
With a delay of 250 milliseconds on that same schedule, the refreshed observation sees the target on the right. Both refreshed and guarded policies can then select it correctly. This explains why timing failures need not grow monotonically for every individual schedule: after a one-time transition, the interface becomes stable again.
Cancellation is not completion
At the 100-millisecond delay, the executed fixture records twelve wrong stale taps, five wrong refreshed taps, and five guarded cancellations with no wrong guarded tap. The script reports correct taps, wrong taps, and cancellations separately for each delay. A guard that cancels every request would have no wrong taps and no useful completions. Its next step would need a bounded re-observation and retry, with explicit authorization for the eventual action. This demonstration stops at the first attempted commit so the contracts remain easy to compare.
Revision validation is an idealized atomic tool feature. Ordinary coordinate-tap interfaces may not provide it. Checking a screenshot in the caller and then tapping still leaves another interval in which the screen can change. The guard's benefit depends on the executor actually enforcing the comparison at the action boundary.
Keep the schedules paired
The raw data records observation time, swap time, commit time, and outcome for each of 240 policy–delay–schedule combinations. Reusing the same twenty schedules removes one avoidable source of comparison noise: a refreshed policy cannot benefit merely because it was given later swaps.
The observed improvement is a property of a specific timing contract. A revision guard is safe here because the executor compares revisions at the same boundary where it would commit the tap. Moving that comparison into an earlier caller-side observation restores the race. Cancellation also remains a distinct outcome; counting it as completion would convert a useful safety mechanism into an inflated task score.
Read the tested Python and recorded output
"""The gap between a fresh screenshot and a safe tap
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
from random import Random
rng = Random(36)
swaps = [rng.choice((25, 75, 125, 275, 525)) for _ in range(20)]
def observe(now, swap):
revision = int(now >= swap)
return {'target_slot': revision, 'revision': revision}
def evaluate(policy, delay):
counts = dict(correct=0, wrong=0, canceled=0)
for swap in swaps:
observed_at = 0 if policy == 'stale' else delay
observation = observe(observed_at, swap)
current = observe(delay + 30, swap)
if policy == 'guarded' and observation['revision'] != current['revision']:
counts['canceled'] += 1
continue
slot = observation['target_slot']
key = 'correct' if slot == current['target_slot'] else 'wrong'
counts[key] += 1
assert sum(counts.values()) == 20
return counts
print('swap_times_ms', swaps)
for delay in (0, 100, 250, 500):
for policy in ('stale', 'refreshed', 'guarded'):
result = evaluate(policy, delay)
print('delay_ms', delay, policy, result)
if policy == 'guarded':
assert result['wrong'] == 0
rows=[]
for delay in (0,100,250,500):
for policy in ('stale','refreshed','guarded'):
for index,swap in enumerate(swaps):
observation=observe(0 if policy=='stale' else delay,swap); current=observe(delay+30,swap)
outcome='canceled' if policy=='guarded' and observation['revision']!=current['revision'] else ('correct' if observation['target_slot']==current['target_slot'] else 'wrong')
rows.append(dict(case=index,swap_ms=swap,delay_ms=delay,policy=policy,observed_ms=0 if policy=='stale' else delay,commit_ms=delay+30,outcome=outcome))
RESULTS=dict(swaps_ms=swaps,cases=rows,totals=[dict(delay_ms=d,policy=p,**evaluate(p,d)) for d in (0,100,250,500) for p in ('stale','refreshed','guarded')])
assert len(rows)==240
print('Matched schedule audit:',len(rows),'records')
Recorded output
swap_times_ms [125, 25, 25, 125, 25, 25, 525, 75, 75, 125, 275, 125, 525, 125, 525, 25, 525, 275, 275, 275]
delay_ms 0 stale {'correct': 15, 'wrong': 5, 'canceled': 0}
delay_ms 0 refreshed {'correct': 15, 'wrong': 5, 'canceled': 0}
delay_ms 0 guarded {'correct': 15, 'wrong': 0, 'canceled': 5}
delay_ms 100 stale {'correct': 8, 'wrong': 12, 'canceled': 0}
delay_ms 100 refreshed {'correct': 15, 'wrong': 5, 'canceled': 0}
delay_ms 100 guarded {'correct': 15, 'wrong': 0, 'canceled': 5}
delay_ms 250 stale {'correct': 4, 'wrong': 16, 'canceled': 0}
delay_ms 250 refreshed {'correct': 16, 'wrong': 4, 'canceled': 0}
delay_ms 250 guarded {'correct': 16, 'wrong': 0, 'canceled': 4}
delay_ms 500 stale {'correct': 0, 'wrong': 20, 'canceled': 0}
delay_ms 500 refreshed {'correct': 16, 'wrong': 4, 'canceled': 0}
delay_ms 500 guarded {'correct': 16, 'wrong': 0, 'canceled': 4}
Matched schedule audit: 240 records