Gemini 2.5 Pro on AAL-D-001
The third complete model evaluation on AAL-D-001, and the first to use multi-run methodology (3 runs × 250 cases = 750 scored observations). Gemini 2.5 Pro achieves 99.6% detection accuracy with perfect dual-exception recall and zero false positives in the clean v1.0 run. A follow-on v1.1 evaluation with explicit operational convention guidance reveals that escalation accuracy was severely underestimated in v1.0 due to a missing policy specification — and that exposure arithmetic is a genuine, persistent weakness.
Protocol
Three independent runs per case at temperature 0 using the Gemini API. The detection interval is a Wilson 95% interval computed by case (the 3 runs of each case grouped). A resume-safe checkpoint system ensured no data loss across API quota interruptions. Errored slots from billing exhaustion were re-run after top-up to produce a fully clean 750-observation dataset.
The escalation gap was a benchmark artifact
v1.0 showed 75.7% escalation accuracy (395/522, eval_out/eval_results_gemini_gemini-2.5-pro.json). Dataset analysis revealed that ground truth escalation is effectively categorical by exception type: all price, quantity, settlement date, counterparty, SSI, booking, and allocation exceptions require escalation; only commission and most duplicate exceptions do not. This rule was absent from the v1.0 prompt. After one paragraph of escalation policy guidance was added, Pro's escalation accuracy in the later full run was 99.0% (517/522, eval_out_v2; the file does not record the prompt version). The v1.0 figure should not be interpreted as measuring model capability.
Exposure arithmetic is a genuine weakness
Exposure accuracy was 80.7% (155/192) on the v1.0 prompt and 76.0% in the later eval_out_v2 run (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases). Correction (2026-10-12): the AAL-D-001 answer key recorded foreign-currency amounts as USD on 11 FX-forward cases: rate differences on 9 forward-rate breaks (147, 149, 153, 160, 162, 163, 167, 174, 175) and notional gaps on 2 notional breaks (148, 178). For example, on case 147 (USD/KRW NDF, $8M notional, 0.05 rate difference) the key records $400,000; the $290.28 the models returned is correct. The earlier statement that models failed this calculation has been removed. AAL-D-001 exposure figures were re-scored on the corrected key (AAL-D-001 v1.0.1, under review in the public repo).