Research / Agent systems
Why normalized clicks miss fixed-margin controls
A viewport-relative point and a right-anchored control obey different transformations. Target width decides when that difference becomes a miss.
A button sits at x = 750 in an 800-pixel-wide window. Widen the window to 1,200 pixels. Where should the click move? The answer depends on the layout rule. If the button stays fifty pixels from the right edge, its new center is 1,150. If it stays at 93.75% of the width, its center is 1,125. “Normalize the coordinates” is not a universal repair.
Inspect a case
Choose the transformation before the coordinate system
OSWorld exposes screenshots and accessibility information for agents acting in real computer environments. The paper also examines sensitivity to window position and size. This example borrows that evaluation concern, but uses a coordinate simulator rather than OSWorld, a vision model, or a desktop automation tool. OSWorld, Sections 2.3–2.4 and 5.2
There are three distinct operations. Absolute replay preserves the old point. Normalized replayPositions expressed as fractions of an image or viewport’s width and height. preserves x/width and y/height. Element-relative replay locates the current element rectangle and chooses the same position inside it, here the center. The last method only helps if locating the elementConnecting a requested object or operation to the correct actionable element. succeeds; giving it the true rectangle in a toy simulator removes that perception problem by assumption.
We create ten layouts: five with a horizontal position proportional to viewport width, and five with a fixed right-edge margin. All buttons remain vertically centered. Each is evaluated at five viewport sizes, giving fifty layout–viewport cases. Buttons have a twenty-pixel width and height. A click succeeds when both coordinate errors are at most ten pixels.
A miss can have a small numerical error
Return to the fifty-pixel margin example. Normalized replay produces x = 750 × 1200 / 800 = 1125. The correct center is 1150, so the horizontal error is twenty-five pixels. That may sound small on a large display, but it lies outside the ten-pixel half-width. Absolute replay misses by four hundred pixels. Element-relative replay hits because it uses the current rectangle.
For proportional layouts, normalized replay exactly preserves the construction rule and therefore hits every case. For fixed-margin layouts, its error grows with the viewport change and the margin. Reporting those families separately is more informative than a single average. Otherwise a test suite dominated by scalable layouts can make normalization look universally reliable.
In the executed fixture, normalized replay hits 25/25 proportional cases but only 5/25 fixed-margin cases. Absolute replay hits 5/25 in each family; the oracle element policy hits 25/25 in both. The code also reports mean Euclidean error for each family and policy. Mean error and hit rate answer different questions: the former measures geometry, while the latter incorporates the target's clickable tolerance. A larger button could improve hit rate without improving coordinate prediction at all.
Keep grounding costs visible
The ideal element policy is an upper-bound geometry baseline, not a free improvement available to every agent. A real interface may have missing labels, stale bounding boxes, occlusion, or duplicate controls. Conversely, a stable semantic element identifier can avoid pixel replay entirely. Compare policies using the information they actually receive, and account for any extra observation needed to obtain a new rectangle.
Target tolerance is a separate variable
The extension holds every predicted point fixed and varies the target half-width from 5 to 80 pixels. Its output records how many of the 25 fixed-margin cases would become hits. Increasing the hit count in that sweep does not improve the transformation: it only makes the same errors less consequential.
For a right margin m, normalized replay from width W₀ to W produces horizontal error m(1 − W/W₀). The sign tells which side of the target is missed; the magnitude grows with both the margin and the relative resize. This closed-form error predicts the plotted points and provides a sharper diagnostic than attributing misses to general agent unreliability.
Read the tested Python and recorded output
"""Why normalized clicks miss fixed-margin controls
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
from math import hypot
VIEWPORTS = [(800, 600), (1200, 900), (1000, 700), (600, 800), (1400, 1000)]
layouts = [(kind, i) for kind in ('proportional', 'right_margin') for i in range(5)]
def center(layout, viewport):
kind, i = layout
width, height = viewport
x = width * (0.2 + 0.1 * i) if kind == 'proportional' else width - 50 - 5 * i
return x, height / 2
for kind in ('proportional', 'right_margin'):
for policy in ('pixel', 'normalized', 'element'):
hits, errors = 0, []
for layout in [x for x in layouts if x[0] == kind]:
old = center(layout, VIEWPORTS[0])
for viewport in VIEWPORTS:
target = center(layout, viewport)
if policy == 'pixel':
click = old
elif policy == 'normalized':
click = old[0] * viewport[0] / 800, old[1] * viewport[1] / 600
else:
click = target # Oracle current element center, not learned grounding.
dx, dy = click[0] - target[0], click[1] - target[1]
hits += abs(dx) <= 10 and abs(dy) <= 10
errors.append(hypot(dx, dy))
print(kind, policy, f'hits={hits}/25', f'mean_error={sum(errors)/25:.1f}px')
assert center(('right_margin', 0), (1200, 900))[0] == 1150
rows = []
for layout in layouts:
for width,height in VIEWPORTS:
target=center(layout,(width,height)); old=center(layout,VIEWPORTS[0])
for policy in ('pixel','normalized','element'):
click=old if policy=='pixel' else ((old[0]*width/800,old[1]*height/600) if policy=='normalized' else target)
dx,dy=click[0]-target[0],click[1]-target[1]
rows.append(dict(kind=layout[0],layout=layout[1],width=width,height=height,policy=policy,target=list(target),click=list(click),dx=dx,dy=dy,error=hypot(dx,dy),hit=abs(dx)<=10 and abs(dy)<=10))
tolerance=[]
for half_width in (5,10,20,40,80):
tolerance.append(dict(half_width=half_width,hits=sum(abs(r['dx'])<=half_width and abs(r['dy'])<=half_width for r in rows if r['kind']=='right_margin' and r['policy']=='normalized')))
RESULTS=dict(cases=rows,target_tolerance=tolerance)
print('Right-margin normalized hit counts by half-width:',tolerance)
Recorded output
proportional pixel hits=5/25 mean_error=153.2px
proportional normalized hits=25/25 mean_error=0.0px
proportional element hits=25/25 mean_error=0.0px
right_margin pixel hits=5/25 mean_error=297.9px
right_margin normalized hits=5/25 mean_error=21.0px
right_margin element hits=25/25 mean_error=0.0px
Right-margin normalized hit counts by half-width: [{'half_width': 5, 'hits': 5}, {'half_width': 10, 'hits': 5}, {'half_width': 20, 'hits': 15}, {'half_width': 40, 'hits': 21}, {'half_width': 80, 'hits': 25}]