Claude Sonnet 4.6 on AAL-D-001
The fifth complete model evaluation on AAL-D-001, and the first from Anthropic. Claude Sonnet 4.6 was run three times across all 250 cases using the same multi-run methodology as the Gemini evaluations. An initial v1.1 run revealed a systematic false positive pattern where Claude flagged the internal status field as an EXC-STAT exception on clean cases, requiring a v1.2 prompt addition. After correction, Claude achieves 98.8% detection with zero false positives, but shows meaningfully lower exposure accuracy (63.5%) and dual-exception recall (87.4%) than both Gemini models.
The EXC-STAT false positive pattern
In the initial v1.1 run, Claude produced a 67.5% false positive rate. All 154 false positives were EXC-STAT: Claude treated the internal status field as an exception because it appeared in the internal record but not the counterparty confirmation. Internal workflow fields like status, book, account, and broker are intentionally absent from counterparty confirmations by dataset design. A one-sentence v1.2 prompt clarification eliminated all false positives. Gemini models did not exhibit this behavior, suggesting Claude has a stronger prior toward treating field-presence asymmetry as a discrepancy.
Exposure accuracy gap
Claude exposure accuracy (64.3%) is 11.7 points below Gemini 2.5 Pro (76.0%) and 15.2 points below Gemini 2.5 Flash (79.5%) on the same later runs (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases). The earlier statement that the same FX forward cases tripped up every model was removed: the answer key mis-recorded those cases.
Dual-exception recall gap
Claude catches both exceptions in dual-exception cases 87.4% of the time, versus 100% for both Gemini models. In roughly 1 in 8 dual-exception cases, Claude identifies the primary exception but misses the secondary one — meaning exception reports may be incomplete on complex multi-break cases.