Models detect exceptions far more reliably than they quantify them
Across Gemini 2.5 Pro and Flash, in both v1.0 and v1.1 prompt configurations, the gap between detecting an exception and correctly quantifying its dollar exposure is large and stable: ~99.5% detection against 76.0–80.7% exposure accuracy (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases). The earlier statement that both models return the same wrong numbers on the same FX forward cases was removed: on 11 FX-forward cases the answer key, not the models, was wrong.
Correction (2026-10-12)
Correction (2026-10-12): the AAL-D-001 answer key recorded foreign-currency amounts as USD on 11 FX-forward cases: rate differences on 9 forward-rate breaks (147, 149, 153, 160, 162, 163, 167, 174, 175) and notional gaps on 2 notional breaks (148, 178). For example, on case 147 (USD/KRW NDF, $8M notional, 0.05 rate difference) the key records $400,000; the $290.28 the models returned is correct. The earlier statement that models failed this calculation has been removed. AAL-D-001 exposure figures were re-scored on the corrected key (AAL-D-001 v1.0.1, under review in the public repo).
Operational implication
Model-generated exposure figures should be treated as indicative rather than authoritative. For triage and screening, detecting the right breaks at 99.5% accuracy is high-value. For prioritization by materiality and regulatory exposure reporting, exposure figures require validation against system-of-record calculations.