Research / Agent systems
What text-only observations erase in visual web tasks
Four identical labels hide four different colors. A contact sheet exposes exactly what the observation representation keeps and loses.
Four product cards all say “Travel mug.” One is red, one blue, one green, and one yellow. Asked for the red mug, a system that receives only those labels does not lack a clever enough search query. It lacks distinguishing evidence. Better reasoning cannot recover a feature that the observation representation removed.
Inspect a case
Construct an information bottleneck
VisualWebArena combines web interaction with tasks requiring visual information. Its observation options include screenshots, accessibility trees, and screenshots with element identifiers. Identifiers help connect a visual object to an action; they do not themselves identify which object meets a visual request. VisualWebArena, Sections 3.1–3.2
The local demonstration builds twenty-four small RGB arrays, organized as six four-card groups. Every card has the same visible text label. Each image is a solid patch representing one of four colors. Group order rotates, so always choosing the first card cannot be rescued by a consistent catalog order. There are twenty-four queries: one request for each color within each group.
A text-only rule selects the first card whose label matches. A pixel rule computes each card's mean RGB and selects the smallest squared distance from the requested color. The arrays are deliberately tiny and generated in memory, so no imaging package, image download, or vision-language model is involved. Solid patches are enough to test whether relevant information reaches a decision rule.
Do the distance calculation
Suppose the requested red is (220, 40, 40), and a blue distractor has mean (40, 70, 220). Squared distance to red is 180² + (−30)² + (−180)² = 65,700. The red card has distance zero. Sorting by this value selects red even when it is not the first result. Squared distance avoids an unnecessary square root while preserving the ranking.
Across the fixture, the first-card rule answers six of twenty-four requests correctly: one per group. The mean-color rule answers all twenty-four correctly. Perfect selection follows from an intentionally separable dataset: each requested RGB value appears exactly on one solid card. The next question is which distinctions the same representation cannot retain.
Separate perception from action grounding
The code returns a card index, not a screen coordinate. That separates choosing the desired visual object from clicking its corresponding control. A full interface must maintain the mapping from image region to current element identifier. Correct color classification attached to the wrong button is still an incorrect action.
Several extensions would break this baseline: a red background around a blue object, two different shapes with identical mean color, or a desired size visible only in text. Those are not implementation bugs; they expose the restricted representation. Combining text and pixels should be evaluated on cases where each contributes necessary information, not only cases designed to favor one modality.
A richer observation can still discard the wrong thing
The additional check constructs two 64-pixel patterns. One begins with 32 red pixels followed by 32 blue pixels; the other reverses that arrangement. Both have mean RGB (130, 55, 130), but their first pixels differ. No rule using only the mean can distinguish those arrangements, regardless of how accurate its arithmetic is.
The solid-card result and the collision test answer complementary questions. Pixels restore the color information missing from the shared labels. Averaging then removes spatial information that a different task might need. Representation quality is therefore task-relative: preserve the feature needed to decide, and retain a stable mapping from the chosen object to the action target.
Read the tested Python and recorded output
"""What text-only observations erase in visual web tasks
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
COLORS = {'red': (220, 40, 40), 'blue': (40, 70, 220),
'green': (40, 180, 70), 'yellow': (220, 210, 40)}
names = list(COLORS)
groups = []
for group in range(6):
order = names[group % 4:] + names[:group % 4]
cards = []
for color in order:
pixels = [COLORS[color] for _ in range(64)]
cards.append(dict(label='Travel mug', pixels=pixels, truth=color))
groups.append(cards)
def mean_rgb(pixels):
return tuple(sum(p[c] for p in pixels) / len(pixels) for c in range(3))
def distance(a, b):
return sum((x - y) ** 2 for x, y in zip(a, b))
text_hits = pixel_hits = 0
for cards in groups:
for target in names:
text_choice = 0
pixel_choice = min(range(4), key=lambda i:
distance(mean_rgb(cards[i]['pixels']), COLORS[target]))
text_hits += cards[text_choice]['truth'] == target
pixel_hits += cards[pixel_choice]['truth'] == target
print('cards', sum(map(len, groups)), 'queries', len(groups) * len(names))
print('text_correct', text_hits, 'pixel_correct', pixel_hits)
print('red_blue_squared_distance', distance(COLORS['red'], COLORS['blue']))
assert (text_hits, pixel_hits) == (6, 24)
rows = []
for group,cards in enumerate(groups):
for target in names:
distances=[distance(mean_rgb(card['pixels']),COLORS[target]) for card in cards]
selected=min(range(4),key=lambda i:distances[i])
rows.append(dict(group=group,target=target,order=[c['truth'] for c in cards],text_choice=0,pixel_choice=selected,distances=distances))
# Same RGB mean, different center: averaging discards spatial arrangement.
red, blue = COLORS['red'], COLORS['blue']
pattern_a = [red]*32 + [blue]*32
pattern_b = [blue]*32 + [red]*32
assert mean_rgb(pattern_a) == mean_rgb(pattern_b)
assert pattern_a[0] != pattern_b[0]
RESULTS=dict(colors=COLORS,queries=rows, correct=dict(text=text_hits,pixels=pixel_hits),collision=dict(mean=list(mean_rgb(pattern_a)),first_pixel_a=list(pattern_a[0]),first_pixel_b=list(pattern_b[0])))
print('Spatial collision:',mean_rgb(pattern_a),'for two different pixel arrangements')
Recorded output
cards 24 queries 24 text_correct 6 pixel_correct 24 red_blue_squared_distance 65700 Spatial collision: (130.0, 55.0, 130.0) for two different pixel arrangements