Research / Agent systems
WebArena’s evaluation lesson: check the final state
A cart message, a URL, and a persisted-state predicate disagree on the same 20 terminal states. The disagreement is the test.
“Added successfully” can be true about the wrong item. A cart URL can be correct while its quantity is wrong. If an evaluator accepts either signal as task completion, it rewards appearances rather than outcomes. Before evaluating a web agent, write down the exact state that would satisfy the user's request.
Inspect a case
Translate intent into a predicate
WebArena evaluates functional outcomes, with task-specific checks that can inspect page content, URLs, APIs, or database state. Its locator-and-check approach separates finding relevant evidence from deciding whether that evidence satisfies the task. The right evidence depends on the task: reaching a URL can be sufficient for navigation, but not necessarily for changing a cart. WebArena, Section 3.2
Our fixed request is “put exactly two blue mugs in the cart, with no other items.” The success predicateChecking whether the resulting environment satisfies the requested outcome. is cart == {'blue-mug': 2}. This intentionally strict equality captures both the positive requirement and the absence of unwanted additions. A weaker test, cart.get('blue-mug') == 2, would miss an extra red mug.
The example constructs five types of outcome, each repeated four times. One has the correct cart and normal confirmation. One has the correct cart but a missing confirmation and an unexpected URL. Three are failures: a wrong color, a wrong quantity, and an extra item. All three failures still display a plausible confirmation and cart URL. These fixtures make the evaluator's assumptions visible rather than relying on obscure accidental edge cases.
Calculate the disagreement
Eight of the twenty cases genuinely succeed. Message matching accepts sixteen cases: four true positives and twelve false positives. It misses the other four valid outcomes. URL matching makes the same errors because the synthetic cases deliberately couple those two superficial cues. The structured predicate accepts the eight correct states and rejects the twelve incorrect ones.
This is not an empirical estimate of how often real shopping sites behave this way. It is an evaluator unit test with adversarially chosen ordinary failure states. A perfect result from the structured checker is expected because these cases exercise the explicit contract it implements. It does not prove the checker covers all possible cart semantics.
The confusion-matrix helper prints true positives, false positives, false negatives, and true negatives. Accuracy alone would obscure what happened: the message checker has only four correct decisions out of twenty, while its claimed-success count looks encouraging. False positives matter especially when a downstream system treats “complete” as permission to stop checking.
Keep the oracle independent
In a larger fixture, hand-label outcomes before writing the evaluator, then review disagreements. Do not generate labels by calling the same predicate you are evaluating; that would guarantee agreement without testing correctness. This code stores explicit labels and separately computes predictions. It also avoids browser automation entirely: the objects represent terminal states, not actions on a real store.
Evaluation should inspect persisted state after any asynchronous update has settled, and should identify the correct user and cart. Those concerns are intentionally absent from this in-memory example. Adding them later should expand the evidence contract, not silently change what success means.
Positive requirements are only half the contract
One additional predicate checks whether the blue-mug quantity equals two but ignores other items. It gets all eight valid carts right, yet also accepts the four extra-item cases. Its confusion matrix is TP=8, FP=4, FN=0, TN=8. This counterexample distinguishes a missing negative requirement from a misleading confirmation message.
State-based evaluation therefore needs a specified state projection. Exact equality is right for this request because “no other items” is explicit. If unrelated items were allowed, exact equality would overreject. The evaluator should encode the user’s actual contract, not use the strictest comparison merely because it is easiest to implement.
Read the tested Python and recorded output
"""WebArena’s evaluation lesson: check the final state
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
TEMPLATES = [
('valid', {'blue-mug': 2}, 'Added successfully', '/cart', True),
('quiet', {'blue-mug': 2}, '', '/catalog', True),
('color', {'red-mug': 2}, 'Added successfully', '/cart', False),
('count', {'blue-mug': 1}, 'Added successfully', '/cart', False),
('extra', {'blue-mug': 2, 'red-mug': 1}, 'Added successfully', '/cart', False),
]
cases = [dict(name=f'{name}-{i}', cart=cart.copy(), message=msg,
url=url, label=label)
for i in range(4) for name, cart, msg, url, label in TEMPLATES]
def confusion(check):
result = dict(TP=0, FP=0, FN=0, TN=0)
for case in cases:
predicted, actual = check(case), case['label']
key = ('TP' if actual else 'FP') if predicted else ('FN' if actual else 'TN')
result[key] += 1
return result
checks = {
'message': lambda c: 'successfully' in c['message'],
'url': lambda c: c['url'] == '/cart',
'state': lambda c: c['cart'] == {'blue-mug': 2},
}
for name, check in checks.items():
print(name, confusion(check))
assert len(cases) == 20
assert confusion(checks['state']) == dict(TP=8, FP=0, FN=0, TN=12)
rows = []
for case in cases:
rows.append(dict(case, predictions={name:check(case) for name,check in checks.items()}))
weak = lambda c: c['cart'].get('blue-mug') == 2
assert confusion(weak) == dict(TP=8,FP=4,FN=0,TN=8)
RESULTS=dict(cases=rows, confusion={name:confusion(check) for name,check in checks.items()}, positive_only=confusion(weak))
print('Positive-only predicate',confusion(weak))
Recorded output
message {'TP': 4, 'FP': 12, 'FN': 4, 'TN': 0}
url {'TP': 4, 'FP': 12, 'FN': 4, 'TN': 0}
state {'TP': 8, 'FP': 0, 'FN': 0, 'TN': 12}
Positive-only predicate {'TP': 8, 'FP': 4, 'FN': 0, 'TN': 8}