An agent benchmark’s headline score is a weighted sum of tasks. Change the task mixture and the headline can move even when every subgroup result stays fixed. GeoKTools makes this easy to miss: its public development set and its withheld test set use the same four K levels, but assign them different proportions.
Research question: what can the released benchmark establish about coverage and scoring before any geospatial agent is run? An audit of 110 public development records finds a concrete composition effect, a tool-availability boundary, and numeric parsing behavior that should be tested before comparing scores.
The pinned Google Research release contains 85 GeoKTools development cases and 25 GeoExpertQueries development cases. The 210 and 76 corresponding test cases are withheld. All 110 development IDs are unique. In GeoKTools, every parsed answer agrees with its raw JSON form. Those are structural checks of the release, not evidence that the answers remain current or that an agent can solve them.
GeoKTools’ K means the number of distinct tool families required by its construction. It is not the number of calls a particular agent executes. The development distribution is 10, 50, 15 and 10 cases for K=1 through K=4. Nineteen template IDs are represented, but one is explicitly marked as an earlier-wording variant of another. Merging only that declared variant yields 18 template families. This is not a claim that a case is invalid; it is a reminder that 85 parameterized questions are not 85 independent task designs.
The release also carries time provenance. Seventy-three GeoKTools answers have the snapshot date September 15, 2026; eleven have September 23; one has September 29. A freshly fetched geographic value can disagree with a frozen answer without implying faulty computation. Conversely, a matching answer does not validate the temporal assumptions of a new application.
Let aₖ be a per-K score between zero and one. Let pₖ and qₖ be the development and held-out weights. The mixture-only difference is the sum of (pₖ − qₖ)aₖ. Its largest possible absolute value is half the sum of the absolute weight differences. That is the total variation distance between the two mixtures, and here it equals 11.204 percentage points.
This is an exact bound, not an estimated performance gap. The code checks all 16 zero/one corners of the four-score cube as an independent verification. The maximum occurs because K=2 is overweighted in development: 58.82% versus 47.62%. A method that is perfect only on K=2 would get those two aggregate scores despite having identical per-K performance in both mixtures.
For a less extreme illustrative profile of 90%, 80%, 60% and 40% across K=1…4, the two weighted means are 72.94% and 70.95%. These rates are deliberately chosen inputs, not Google or author measurements. Reporting per-K scores alongside the overall mean makes this mixture effect visible. Standardizing to common weights can answer a comparison question, but it cannot repair differences in the underlying test questions.
The four sliders are hypothetical per-K scores. Only the released development weights and documented held-out weights are factual inputs.
GeoExpertQueries asks a different question. Its reference material describes analytical workflows rather than a short list of exact outputs. Ten of the 25 released tasks are labeled solvable with the source toolset and fifteen beyond it. “Beyond current toolset” does not mean scientifically impossible, and it does not predict what another agent with different data access can accomplish.
The four domain counts are 11 hazards/resilience, 5 public health, 8 site selection/urban planning and 1 operational/commercial. The reference workflows contain 13–40 analysis list items, with a median of 19. Those are authored bullets, not timed steps, executed tools or comparable units of reasoning. Counting them is useful for inspecting reference complexity, but not for claiming that one task takes twice as much work as another.
This split calls for two reports: a coverage report explaining what data and tools were available, and a quality report explaining whether the resulting analysis was valid. A scientifically appropriate abstention should not be conflated with a wrong numerical answer. The published evaluation protocol uses a model judge for the workflows, with three ratings on a 0/0.5/1 scale. This audit does not call that judge.
The repository supplies a deterministic alternative to the report’s exact-match-then-LLM-judge protocol. Its numeric branch awards full credit within 10% relative error and otherwise uses max(1 − relative error, 0). A zero-valued reference is a special case: only zero gets credit. Strings use a separate ROUGE-L route, which is outside the numeric tests here.
For a reference of 100, a prediction of 110 scores 1.0; 110.01 scores 0.8999. The score therefore has a discontinuity at the tolerance boundary. This may be an intentional acceptance policy, but it means a tiny numerical change can move a case by roughly a tenth of a point. Seven numeric answer slots in the release are zero, so the zero special case is not merely theoretical.
The parser also distinguishes numbers from number-like strings. A JSON numeric value 1000 scores 1 against reference 1000. The string “1e3” is parsed as 1 by the numeric extraction regex and scores 0.001; “1,000” does the same. The eight local probes cover the boundary, zero, a negative reference and those two formatting cases. They verify code behavior on new test values, not the robustness of an entire agent pipeline.
Of 179 flattened answer slots, 113 are numeric and 66 are strings. Five numeric slots are negative. A global relative tolerance has no knowledge of coordinate systems, distance units or the consequence of a sign error. An operational evaluator should check unit and coordinate conventions before assigning numerical partial credit, and explicitly identify which scoring protocol generated the result.
At minimum: the split, pinned benchmark revision, per-K or per-domain denominators, the agent’s available tool manifest, answer snapshot dates, missing-response policy and scoring protocol. If a model judge is used, include its model/configuration, prompt, number of runs and the aggregation rule. Repeating judge calls measures one source of variation; it does not make a development split representative of all geospatial work.
Uncertainty also needs the right unit. Parameterized cases share templates. A case bootstrap answers a different question from a template-cluster bootstrap. Neither interval alone captures domain shift or dependence on shared data providers. The public record supplies enough structure to investigate those choices, but the correct uncertainty model depends on the population the score is intended to describe.
The attached input is intentionally a metadata projection: IDs, labels, snapshot dates and structural counts. It omits prompts, workflow prose, geographic values and all golden answers. This matters because the repository’s CC BY 4.0 data license explicitly excludes Google Maps content and preserves third-party provider terms. The audit remains reproducible from the projection without redistributing those values.
The included metadata projection is enough to recompute the 110-record census, mixture bound and numeric probes offline. An optional preparation script can rebuild that projection from separately obtained, hash-verified originals without copying answer values into its output.