Research / Agent systems
Memory needs a scope: a rule-change test inspired by Reflexion
A 30-episode trace shows why the conditions attached to a failure note matter more than the size of the notebook.
A failure note can be accurate when written and misleading when reused. Imagine a parcel-routing task in which lane A is temporarily closed. After a rejection, the system records “avoid A.” That note is useful until maintenance ends and lane B closes instead. An ever-growing list of prohibitions can eventually forbid every available lane.
Inspect a case
Memory is an input to a policy
Reflexion separates an actor, an evaluator, and a self-reflection component. Feedback from a trial becomes text available to later trials; the model's weights need not change. The paper bounds its memory buffer, usually to one to three experiences. Permanent storage and version-tagged notes are the alternatives proposed here, not descriptions of the original method. Reflexion, Section 3
The teaching fixture replaces language generation with explicit note templates. A note contains the lane that failed and the rule version under which it failed. The policy selects the first lane not prohibited by applicable notes. This deliberate simplicity makes memory selection inspectable: there is no hidden model reasoning to explain away a surprising result.
Thirty episodes use two rule versions. In episodes 1–15, only B is valid. In episodes 16–30, only A is valid. The version number is provided as part of the current observation. That information is an assumption of the experiment, not an ability the policy somehow infers.
Follow the notes through the boundary
With no memory, the fixed preference for A fails fifteen times and succeeds fifteen times. Permanent memory fails on episode 1, records “avoid A,” and then correctly uses B for fourteen episodes. At episode 16, B fails and the buffer gains “avoid B.” Both lanes are now prohibited. The policy abstains for the remaining fourteen episodes, leaving fourteen successes overall.
Versioned memory also fails once initially, then succeeds fourteen times. At the boundary, the old “avoid A” note does not apply to version 2. The default A is therefore selected and succeeds for all fifteen remaining episodes, producing twenty-nine successes. The difference comes from retrieval scope, not greater intelligence or a more persuasive reflection.
Report the abstentions separately. An agent that stops acting because its notes contradict one another is different from one that repeatedly chooses an invalid lane. Both are failures to complete the task, but their remedies differ. The first needs conflict resolution; the second may need better observations or policy repair.
Use a memory contract
A practical note schema should identify the observation supporting it, the action attempted, the actual feedback, and the conditions under which the lesson applies. “Avoid A under routing version 1” is a narrower claim than “A is always wrong.” Versioning is attractive when the environment exposes reliable revisions. When it does not, timestamps and confidence decay are weaker substitutes, not equivalent guarantees.
The demonstration intentionally makes permanent prohibitions brittle. It does not establish that all long memory is harmful: a policy capable of revising contradictory notes could recover. Nor does it test whether an LLM can write the right conditional lesson. Its value is a reproducible counterexample to treating accumulated advice as timeless truth.
Read the transition, not only the total
The downloadable records retain the active prohibitions, chosen lane, actual valid lane, and outcome for every policy and episode. At episode 16, permanent memory selects B from its earlier advice and learns a second prohibition. Episode 17 is the decisive transition: neither lane remains eligible, even though A is available. The versioned policy has no applicable prohibition in version 2 and uses A immediately.
The useful invariant is not “more recent memories win.” It is “a note can constrain the policy only while its stated conditions apply.” A short buffer can incidentally forget stale advice, but eviction and applicability are different mechanisms. If a recurring rule returns later, scoped memory can retrieve an old valid lesson without forcing it onto the intervening world.
Read the tested Python and recorded output
"""Memory needs a scope: a rule-change test inspired by Reflexion
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
EPISODES = [(1 if i < 15 else 2, 'B' if i < 15 else 'A')
for i in range(30)]
def evaluate(mode):
notes = []
wins = invalid = abstained = 0
history = []
for version, correct in EPISODES:
applicable = [n for n in notes if mode == 'permanent'
or (mode == 'versioned' and n['version'] == version)]
blocked = {n['avoid'] for n in applicable}
lane = next((x for x in ('A', 'B') if x not in blocked), None)
success = lane == correct
wins += success
if lane is None:
abstained += 1
elif not success:
invalid += 1
if mode != 'none':
notes.append(dict(version=version, avoid=lane))
history.append(int(success))
assert wins + invalid + abstained == 30
return wins, invalid, abstained, sum(history[:15]), sum(history[15:])
print('policy: wins invalid abstained first_half second_half')
for mode in ('none', 'permanent', 'versioned'):
print(mode, *evaluate(mode))
assert evaluate('versioned')[0] == 29
assert evaluate('permanent')[2] == 14
rows = []
for mode in ('none','permanent','versioned'):
notes = []
for episode,(version,correct) in enumerate(EPISODES,1):
active = [n for n in notes if mode == 'permanent' or (mode == 'versioned' and n['version'] == version)]
blocked = {n['avoid'] for n in active}
lane = next((x for x in ('A','B') if x not in blocked),None)
outcome = 'abstained' if lane is None else ('completed' if lane==correct else 'invalid')
rows.append(dict(policy=mode,episode=episode,version=version,correct=correct,choice=lane,blocked=sorted(blocked),outcome=outcome))
if outcome=='invalid' and mode!='none': notes.append(dict(version=version,avoid=lane))
assert sum(r['outcome']=='completed' for r in rows if r['policy']=='versioned')==29
RESULTS=dict(episodes=rows, totals={p:dict(zip(('completed','invalid','abstained','first_half','second_half'),evaluate(p))) for p in ('none','permanent','versioned')})
print('Episode-level audit:',len(rows),'policy–episode records')
Recorded output
policy: wins invalid abstained first_half second_half none 15 15 0 0 15 permanent 14 2 14 14 0 versioned 29 1 0 14 15 Episode-level audit: 90 policy–episode records