Research / Agent systems
Structured edit errors make recovery programmable
A transactional editor can preserve source state and expose a repairable error. Syntax acceptance still says nothing about intended behavior.
“Edit failed” leaves several explanations open. The requested lines might not exist, the replacement might be invalid Python, or the edit might have been applied to the wrong file. A useful agent–computer interface should narrow that uncertainty without forcing the caller to reconstruct the entire environment after every mistake.
Inspect a case
Make edits transactional
SWE-agent treats the interface itself as an important design variable: commands, file views, and feedback are shaped for the agent. Its editor returns relevant context, checks selected errors, and discards invalid edits rather than leaving the repository in a broken intermediate state. The following fixture explores that contract with Python's parser and scripted repair rules. SWE-agent, Sections 2–3
Five disposable files are represented as strings in a dictionary. Each contains a function definition and a return statement. For every file, we attempt two bad edits: a line range outside the two-line file, and a replacement of the function header that omits its colon. That gives ten controlled failure cases. No user's repository is read or changed.
The edit function first validates the one-based inclusive line range. Only then does it assemble a candidate string and pass it to ast.parse. If parsing fails, it returns a syntax category and parser line number. If either check fails, the stored file remains unchanged. Successful parsing permits the candidate to replace the original.
Feedback must enable an action
For an invalid range, the structured response includes the current line count. The scripted repair changes the request to line 1. For a syntax error, it uses the returned category to restore the known colon. Each case then succeeds on its second attempt. The generic-feedback policy receives only an undifferentiated error and repeats its first request, so neither of its two attempts succeeds.
This comparison is intentionally narrow. The structured policy is programmed to exploit exactly these error categories, while the generic policy has no diagnostic action. Its failure is not evidence that an LLM cannot infer a repair from a vague message. It demonstrates that an interface can make recovery mechanically available rather than leaving it dependent on inference.
Two attempts per case means twenty edit calls for either policy. Structured feedback yields ten accepted edits; generic feedback yields zero. The shared action budget matters: comparing one policy after retries against another after only one attempt would confuse feedback quality with extra opportunity.
Syntax is a guardrail, not a correctness proof
A candidate returning 999 instead of 0 parses perfectly. The parser protects source structure, not intended behavior. A production-quality workflow therefore needs a separate semantic test stage, with clear reporting of which checks were run. A successful edit response should never be worded as though it proved the issue was fixed.
Error messages should include enough context to repair a request, but not unlimited file dumps. Here, category, line number, and allowed range are sufficient because the fixture is tiny. Larger repositories require careful context windows and stable file revisions; an otherwise valid range can become stale when another edit inserts lines above it.
Exercise the gap between syntax and semantics
The extension replaces the return statement in f0 with return 999. The transaction accepts the edit because the source parses. Inspecting the resulting syntax tree yields 999 where the fixture’s function contract requires 0. The semantic check fails even though the edit itself succeeded.
An interface should expose both statuses without collapsing them into a single “success.” Accepted means the requested state change met the editor’s structural contract. Correct means independent behavioral checks agree with the intended change. Keeping those concepts separate makes it possible to retry a rejected edit without mistaking an accepted edit for a finished repair.
Read the tested Python and recorded output
"""Structured edit errors make recovery programmable
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
import ast
BASE = {f'f{i}.py': f'def f{i}():\n return {i}\n' for i in range(5)}
def edit(files, name, start, end, replacement):
lines = files[name].splitlines()
if not 1 <= start <= end <= len(lines):
return {'error': 'range', 'lines': len(lines)}
candidate = '\n'.join(lines[:start-1] + [replacement] + lines[end:]) + '\n'
try:
ast.parse(candidate)
except SyntaxError as exc:
return {'error': 'syntax', 'line': exc.lineno}
files[name] = candidate
return {'ok': True}
for structured in (False, True):
accepted = calls = 0
for i, name in enumerate(BASE):
for mistake in ('range', 'syntax'):
files = BASE.copy()
start = 99 if mistake == 'range' else 1
replacement = f'def f{i}():' if mistake == 'range' else f'def f{i}()'
for attempt in range(2):
before = files[name]
result = edit(files, name, start, start, replacement)
if not structured and not result.get('ok'):
result = {'error': 'edit_failed'}
calls += 1
if result.get('ok'):
accepted += 1
break
assert files[name] == before
if structured:
if result['error'] == 'range':
start = min(1, result['lines'])
elif result['error'] == 'syntax':
replacement = f'def f{i}():'
print('structured=' + str(structured), 'accepted', accepted, 'calls', calls)
assert accepted == (10 if structured else 0)
rows=[]
for structured in (False,True):
for i,name in enumerate(BASE):
for mistake in ('range','syntax'):
files=BASE.copy(); start=99 if mistake=='range' else 1
replacement=f'def f{i}():' if mistake=='range' else f'def f{i}()'
for attempt in (1,2):
before=files[name]; response=edit(files,name,start,start,replacement)
if not structured and not response.get('ok'): response={'error':'edit_failed'}
rows.append(dict(policy='structured' if structured else 'generic',file=name,mistake=mistake,attempt=attempt,response=response,unchanged=files[name]==before))
if response.get('ok'): break
assert files[name]==before
if structured:
if response['error']=='range': start=1
else: replacement=f'def f{i}():'
files=BASE.copy()
semantic_response=edit(files,'f0.py',2,2,' return 999')
assert semantic_response.get('ok') and files['f0.py'] != BASE['f0.py']
parsed=ast.parse(files['f0.py'])
semantic_value=parsed.body[0].body[0].value.value
assert semantic_value==999
RESULTS=dict(attempts=rows,semantic_counterexample=dict(accepted_by_parser=True,expected_return=0,actual_return=semantic_value,semantic_test_passed=semantic_value==0))
print('Valid syntax / wrong behavior:',semantic_value,'!=',0)
Recorded output
structured=False accepted 0 calls 20 structured=True accepted 10 calls 20 Valid syntax / wrong behavior: 999 != 0