What text-only observations erase in visual web tasks notebook

Python starts when you choose Run.

What text-only observations erase in visual web tasks

Four identical labels hide four different colors. A contact sheet exposes exactly what the observation representation keeps and loses.

Synthetic RGB-feature teaching example; no VLM, screenshot parser, or VisualWebArena evaluation.

"""What text-only observations erase in visual web tasks
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
COLORS = {'red': (220, 40, 40), 'blue': (40, 70, 220),
          'green': (40, 180, 70), 'yellow': (220, 210, 40)}
names = list(COLORS)
groups = []
for group in range(6):
    order = names[group % 4:] + names[:group % 4]
    cards = []
    for color in order:
        pixels = [COLORS[color] for _ in range(64)]
        cards.append(dict(label='Travel mug', pixels=pixels, truth=color))
    groups.append(cards)

def mean_rgb(pixels):
    return tuple(sum(p[c] for p in pixels) / len(pixels) for c in range(3))

def distance(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b))

text_hits = pixel_hits = 0
for cards in groups:
    for target in names:
        text_choice = 0
        pixel_choice = min(range(4), key=lambda i:
            distance(mean_rgb(cards[i]['pixels']), COLORS[target]))
        text_hits += cards[text_choice]['truth'] == target
        pixel_hits += cards[pixel_choice]['truth'] == target
print('cards', sum(map(len, groups)), 'queries', len(groups) * len(names))
print('text_correct', text_hits, 'pixel_correct', pixel_hits)
print('red_blue_squared_distance', distance(COLORS['red'], COLORS['blue']))
assert (text_hits, pixel_hits) == (6, 24)

rows = []
for group,cards in enumerate(groups):
    for target in names:
        distances=[distance(mean_rgb(card['pixels']),COLORS[target]) for card in cards]
        selected=min(range(4),key=lambda i:distances[i])
        rows.append(dict(group=group,target=target,order=[c['truth'] for c in cards],text_choice=0,pixel_choice=selected,distances=distances))
# Same RGB mean, different center: averaging discards spatial arrangement.
red, blue = COLORS['red'], COLORS['blue']
pattern_a = [red]*32 + [blue]*32
pattern_b = [blue]*32 + [red]*32
assert mean_rgb(pattern_a) == mean_rgb(pattern_b)
assert pattern_a[0] != pattern_b[0]
RESULTS=dict(colors=COLORS,queries=rows, correct=dict(text=text_hits,pixels=pixel_hits),collision=dict(mean=list(mean_rgb(pattern_a)),first_pixel_a=list(pattern_a[0]),first_pixel_b=list(pattern_b[0])))
print('Spatial collision:',mean_rgb(pattern_a),'for two different pixel arrangements')

cards 24 queries 24
text_correct 6 pixel_correct 24
red_blue_squared_distance 65700
Spatial collision: (130.0, 55.0, 130.0) for two different pixel arrangements

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook