Research / Agent systems
A fixed seed is not a reset contract
A cached choice survives a partial reset, changes every starting fingerprint, and breaks half the repeated tasks.
A fixed random seed does not imply a fixed starting world. A browser can retain cookies, a server can retain a changed record, and a client can retain a cached selection. If those survive between trials, the same policy and seed can produce different outcomes for reasons unrelated to model quality.
Inspect a case
Reset is a state contract
BrowserGym presents a reset-and-step interaction interface, but its ecosystem also distinguishes backend preparation. Some supported benchmarks require restoring server state before evaluating another agent. AgentLab records reproducibility information and manages experiment execution around these constraints. The important lesson is not that one reset call magically cleans every layer, but that the relevant layers must be identified. BrowserGym, Sections 3–5
Our pure-Python fixture represents three layers: a cookie dictionary, a visit counter, and a cached selection. The task seed determines a target, either left or right. A scripted policy uses a cached choice if one exists; otherwise it selects the current target and caches that choice. Success requires selecting the target on the first visit.
A complete reset reconstructs the entire state from the task seed. A partial reset clears cookies and the counter but retains the selection cache. This is not a claim about BrowserGym's implementation. It is a deliberately faulty reset procedure whose consequences can be checked without launching a browser.
Repeat the exact task after contamination
For each of ten seeds, the first run begins from a complete reset and succeeds. Before the second run, the fixture adds a cookie and extra visits. On odd-numbered seeds it also changes the cached selection to the other side. Even-numbered seeds retain the correct cached choice. These edits represent controlled leftovers, not malicious content or activity on a live account.
Complete reset restores identical starting fingerprints and succeeds on all ten repeats. Partial reset succeeds on only the five even-numbered cases. Its initial fingerprint differs in all ten cases, because even a correct retained cache is different from the specified empty cache. Thus state mismatch is a warning signal, not identical to task failure.
The fingerprint is a SHA-256 digest of a canonical JSON serialization with sorted keys. Canonicalization matters: otherwise dictionary ordering can create a different byte sequence for semantically identical state. Hashes are useful for comparison, but not for diagnosis. When a mismatch appears, inspect a field-level diff rather than staring at two opaque strings.
Fingerprint what can affect the trajectory
A hash only covers the state included in its input. If the real environment depends on a clock, filesystem contents, service version, or server-side preference that was omitted, matching hashes do not establish equivalence. Define the reset contract before building the fingerprint and include a version for that contract.
Full reconstruction is easy for this three-field simulator. Real environments may need database snapshots, fresh browser contexts, or task-specific cleanup. Parallel workers also need isolation: one trial can corrupt another even if both started clean. Those are separate engineering concerns from merely repeating the pseudorandom seed.
Use the hash to detect, then the fields to explain
The extended records retain both starting fingerprints and the field-level differences for each repeat. Every partial-reset mismatch is isolated to cache. On an even seed that retained choice happens to be correct, so the task succeeds despite the mismatch. On an odd seed it points to the opposite side and the repeat fails.
This is why fingerprint equality and task success should be reported separately. The first tests conformity to a starting-state contract. The second tests behavior from that state. A successful contaminated run does not certify the reset procedure, and a differing fingerprint does not by itself explain the failed action. The field-level diff supplies the causal link.
Read the tested Python and recorded output
"""A fixed seed is not a reset contract
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
import hashlib
import json
def fresh(seed):
return dict(seed=seed, target=seed % 2, cookies={}, visits=0, cache=None)
def fingerprint(state):
raw = json.dumps(state, sort_keys=True, separators=(',', ':')).encode()
return hashlib.sha256(raw).hexdigest()
def run(state):
choice = state['cache'] if state['cache'] is not None else state['target']
state['cache'] = choice
state['visits'] += 1
return choice == state['target'] and state['visits'] == 1
for mode in ('partial', 'complete'):
identical = successes = 0
for seed in range(10):
state = fresh(seed)
initial = fingerprint(state)
assert run(state)
state['cookies']['previous_run'] = True
state['visits'] += 7
if seed % 2:
state['cache'] = 1 - state['target']
if mode == 'complete':
state = fresh(seed)
else:
state['cookies'] = {}
state['visits'] = 0
identical += fingerprint(state) == initial
successes += run(state)
print(mode, 'identical_initial_states', identical, 'repeat_successes', successes)
assert (identical, successes) == ((10, 10) if mode == 'complete' else (0, 5))
records=[]
for mode in ('partial','complete'):
for seed in range(10):
state=fresh(seed); initial=fingerprint(state); run(state)
state['cookies']['previous_run']=True; state['visits']+=7
if seed%2: state['cache']=1-state['target']
contaminated=json.loads(json.dumps(state))
if mode=='complete': state=fresh(seed)
else: state['cookies']={}; state['visits']=0
before=json.loads(json.dumps(state)); repeated=fingerprint(state)
changes=[key for key in state if state[key]!=fresh(seed)[key]]
success=run(state)
records.append(dict(mode=mode,seed=seed,initial_hash=initial,repeat_hash=repeated,identical=initial==repeated,contaminated=contaminated,after_reset=before,different_fields=changes,success=success))
RESULTS=dict(repeats=records,contract_fields=sorted(fresh(0)),contract_version=1)
assert all(r['different_fields']==['cache'] for r in records if r['mode']=='partial')
print('All partial-reset mismatches isolated to: cache')
Recorded output
partial identical_initial_states 0 repeat_successes 5 complete identical_initial_states 10 repeat_successes 10 All partial-reset mismatches isolated to: cache