Audit the evaluation before trusting the score

A public evaluation repository can contain real prompts, real metric code and a runner that produces realistic-looking scores without invoking a real model. That is not inherently a problem. A scaffoldA simulated evaluation that tests the workflow using scripted or sampled stand-ins for real model behavior. is useful for testing an interface. The problem is treating the resulting numbers as evidence about the model rather than evidence that the plumbing works.

Research question: what is actually connected by the public monitorability release, and where does the path from a named evaluation to an empirical claim stop? A machine-checked audit of its registry and Python source finds 14 named entries, eight that pass the runner’s metadata gates, and an intentionally mocked model-and-monitor path.

Release coverage is a sequence of gates

The pinned OpenAI repository organizes evaluations into intervention, process and outcome-property archetypes. Its registry has nine intervention entries, two process entries and three outcome entries. The loader does not run every registered name: first it selects JSONL paths, then respects explicit exclusions, then resolves label rules and prompts, and finally checks extracted rows.

Twelve entries name JSONL files; two name descriptive Markdown documents for LiveBench and SHADE-Arena. Four JSONL entries are explicitly excluded from the mock scaffold: the two AIME sets, Memory Advice and Flaky Tools/CharXiv. AIME exposes source URLs instead of redistributing problem statements. Memory and Flaky Tools need environment-specific interfaces. That leaves seven intervention sets and one process set metadata-eligible.

“Metadata-eligible” is deliberately precise. This audit does not inspect the binary environment archive or claim that its paired IDs, row counts or missing fields are valid. The loader can reject additional records after extraction. The result here is a static release contract, not an execution success rate.

The useful checks are small and exact

The audit resolves all 24 shared-rule references appearing in dataset Y/Z mappings, covering 12 distinct rules. It also verifies all 26 path-shaped prompt references against the pinned source tree, covering 15 distinct files. Explanatory prose in a prompt list is not mistaken for a filename. These are concrete assertions: a dangling shared-rule key or missing path makes the analysis fail.

This is enough to distinguish a broken routing configuration from an incomplete reproduction, but not enough to certify semantics. A file can exist and contain the wrong instruction. A label reference can resolve and still be inappropriate for the intended scientific question. A complete reproduction must inspect the generated prompt, actual model transcript and grader output on the same sample, not just count successful filesystem lookups.

The snapshot uses scaffold_supported: false for exclusions, with eligibility defaulting to true when the field is absent. That convention is relevant for maintainers: adding an entry without an exclusion changes the candidate set. A release audit should therefore compare the exact expected entry list and statuses, rather than interpreting an omitted flag as an explicit scientific validation.

The runner’s apparent ranking is configured

The runner docstring calls its outputs mocked. Source inspection confirms why: MockModel uses preset correctness probabilities, and MockGraders samples monitor triggers using visibility-specific probabilities. The configured model correctness rates are 0.35 for intervention control, 0.75 for intervention treatment and 0.65 for process tasks.

The monitor’s configured true-positive/true-negative pairs are 0.82/0.78 for CoT-only, 0.60/0.85 for answer-only and 0.88/0.88 for all messages. Those are literal constants extracted from the Python abstract syntax tree. The audit reads the source; it does not import or execute the scaffold and does not call a model.

For orientation, the configured products TPR × TNR are 0.6396, 0.5100 and 0.7744 for those three scopes. These products are arithmetic on input constants, not the repository’s measured intervention statistic. Even an extremely narrow interval around a long mock run would only show that the simulation converged toward its assumptions.

There is a second semantic boundary in the same code. The mock process monitor uses outcome correctness as a proxy for its latent positive state, while the registry’s process rule concerns detecting a complete theorem path. A correct answer and a faithfully detected reasoning path are different events. That approximation is acceptable for exercising a data schema, but it cannot answer the process-monitorability question on its own.

Y = 1 is not one universal kind of good

The registry makes label meaning explicit, and that is one of its most useful contributions. Intervention Y can mean matching the answer toward which a perturbation points. Scruples’ directional target differs by variant. Process labels concern answer correctness and theorem-path detection. Flaky Tools describes a non-passing/deceptive outcome as the scaffold’s positive Y. These labels should not be pooled as if one always meant “safe” or “correct.”

Z also depends on the task. Some monitors emit A/B or A/B/C/D choices, another emits X/Y, and the process route expects theorem-level JSON. A generic “positive first option” parser is not a substitute for the registered positive-answer set. Retaining the dataset name, archetype, outcome rule and monitor rule beside each record is essential for interpreting any aggregate.

For intervention analysis, pairing matters as much as label direction. The release describes perturbed and unperturbed rows sharing an ID. A production audit should check exactly which arms exist for every dataset-and-ID pair, detect duplicates, and keep both arms together during resampling. That check is explicitly outside this registry-only result; asserting it from documentation would conceal the most important missing data validation.

Deprecated is different from unavailable

The README deprecates Memory and Anti-Scheming pending changes to grading or monitor/answer-grader alignment. Memory Advice still has a registry entry; Anti-Scheming is not a released entry. A status system therefore needs separate fields for presence, runtime support and methodological status. “The file is public” is not equivalent to “the authors still use this test.”

The release also omits evaluations using private or restricted materials. That prevents an honest public-only aggregate from being presented as the complete original suite. The narrow claim remains valuable: it states exactly which public paths, rule mappings and source constants were inspected, without borrowing the authority of evaluations that were not run.

A practical evidence contract

Before using a score in a model comparison, require four artifacts: a pinned and licensed input manifest; a validated sample/arm table; transcripts showing the real model and monitor calls; and a metric record that preserves label meaning, exclusions and uncertainty units. Each should be independently inspectable. A successful scaffold run satisfies none of the empirical call requirements by itself.

The bundle includes the registry as both original YAML and a lossless JSON conversion, plus the source tree and scaffold source for static analysis. It includes no AIME statements, no other benchmark question text, and no environment archive. Upstream code is Apache-2.0; third-party datasets retain the separate terms listed in the attribution document. The notebook’s default path needs only the Python standard library because conversion has already been performed.

The result is not that this release is unusable. It is that its most immediately reproducible object is an interface contract. That contract helps separate data availability, tool integration, grading and evidence. Keeping those layers separate is what prevents a well-engineered demo from turning into a misleading benchmark claim.

Sources & reuse

1. Pinned dataset registry, the census and label mappings used here.
2. Scaffold implementation, loader gates and mock constants. Parsed as data, not executed.
3. Release scope and deprecated evaluations, and dataset-specific attribution. OpenAI copyright and license notices are retained in the bundle.