Benchmark specification gaps can masquerade as model capability differences
The v1.0 evaluation showed Gemini 2.5 Pro with 75.7% escalation accuracy (395/522), which appeared to be a capability weakness. It traced to a missing escalation policy in the prompt: ground truth treated nearly all exception types as requiring escalation, but the prompt gave no guidance on this convention, and Pro, reasoning conservatively, under-escalated. After one paragraph of explicit policy guidance was added, Pro reached 99.0% (517/522) and Flash 98.1% in the later eval_out_v2 run (prompt version not recorded). A v1.0 comparison with Flash was removed on Oct 12, 2026 because no full v1.0 Flash results file could be found.
The mechanism
Ground truth escalation_required was effectively categorical by exception type — a rule that any experienced operations analyst would know implicitly but that was never stated in the benchmark prompt. Models were expected to infer the rule from context. Pro inferred conservatively (low apparent severity → no escalation), which was not wrong given the information provided. The benchmark was measuring inference about unstated rules, not escalation judgment.
Implication for benchmark design
Published benchmark scores should include prompt sensitivity testing, not just a single prompt result. A model that scores poorly on an underspecified benchmark may be more cautious and less prone to hallucinated conventions — a property that looks like a weakness in evaluation but is a strength in deployment. AAL will test prompt sensitivity as a standard part of the evaluation methodology going forward.