Prompt Sensitivity Analysis: v1.0 vs v1.1 on AAL-D-001
A comparative analysis of Gemini 2.5 Pro and Flash performance across two prompt versions: v1.0 (baseline) and v1.1 (with explicit exposure calculation rules and escalation policy). The analysis was motivated by unexpectedly low escalation accuracy for Pro in v1.0. The findings demonstrate that benchmark specification quality is as important as model selection — a missing policy paragraph cost Pro 23.3 points of escalation accuracy — while the exposure results were re-scored after an answer-key error was found.
What changed between v1.0 and v1.1
Two additions: an explicit exposure calculation rule (notional × absolute rate difference for price breaks; notional difference for quantity breaks) and an escalation policy paragraph clarifying that nearly all exception types require escalation, with commission discrepancies and likely duplicates as the primary exceptions. Nothing else in the prompt template changed.
Finding 1: The escalation gap was entirely a specification artifact
Pro's v1.0 escalation accuracy was 75.7% (395/522). Dataset analysis showed the ground truth rule was nearly categorical by exception type, but this rule was absent from the prompt, and Pro under-escalated. With explicit policy guidance, in the later eval_out_v2 run (prompt version not recorded), Pro reached 99.0% (517/522) and Flash 98.1% (512/522). The low v1.0 figure measured prompt underspecification, not model capability.
Finding 2: Exposure results re-scored
Pro's exposure accuracy was 80.7% (155/192) on the v1.0 prompt and 76.0% in the later eval_out_v2 run; Flash reached 79.5% (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases). Correction (2026-10-12): the AAL-D-001 answer key recorded foreign-currency amounts as USD on 11 FX-forward cases: rate differences on 9 forward-rate breaks (147, 149, 153, 160, 162, 163, 167, 174, 175) and notional gaps on 2 notional breaks (148, 178). For example, on case 147 (USD/KRW NDF, $8M notional, 0.05 rate difference) the key records $400,000; the $290.28 the models returned is correct. The earlier statement that models failed this calculation has been removed. AAL-D-001 exposure figures were re-scored on the corrected key (AAL-D-001 v1.0.1, under review in the public repo).
Methodological implication
Any benchmark that does not test prompt sensitivity risks attributing specification gaps to model capability differences. The AAL methodology of iterating prompts and publishing both versions is designed to make this distinction explicit. v1.0 and v1.1 results are not directly comparable and are published separately to preserve the methodological record.
Sources and correction (Oct 12, 2026)
Pro v1.0: eval_out/eval_results_gemini_gemini-2.5-pro.json. Pro and Flash later run: eval_out_v2 results files (aal-eval-platform; prompt version not recorded). A previously shown v1.0 comparison with Flash (94.8% escalation, ~73% exposure) was removed because no full v1.0 Flash results file could be found.