Claude Sonnet 4.6 on AAL-D-002
The second complete evaluation on AAL-D-002, and the first from Anthropic on the margin call dispute benchmark. Claude Sonnet 4.6 was run three times across all 250 cases using the same matched prompt (v1.0) and deterministic scorer as the GPT-4o evaluation. Detection and classification are near-ceiling; amount accuracy at 80.4% edges GPT-4o on every arithmetic metric. Critically, neither model produced a single false positive across 1,128 combined clean-call runs. A methodology finding emerged: strict JSON-only output constraints combined with tight token budgets suppress model reasoning on hard multi-CSA netting cases, with the model choosing to reason in prose before answering rather than comply with format — and answering correctly once given the space.
Protocol
Three independent runs per case at temperature 0 using the Anthropic API (claude-sonnet-4-6). Matched prompt v1.0, identical to the GPT-4o evaluation. A resume-safe checkpoint system with fatal-error handling (billing and auth errors exit immediately; only transient errors retry) ensured zero contamination. Per-call token usage was captured across all runs: 1.17M input and 179K output tokens total, ~$8–10 in API spend.
Amount accuracy edges GPT-4o across all arithmetic metrics
Claude Sonnet leads GPT-4o on all three amount metrics: correct call amount (80.4% vs 76.1%), disputed amount (78.5% vs 76.6%), and undisputed amount (88.5% vs 84.4%). GPT-4o leads on classification and routing: category (94.9% vs 92.2%), field ID (87.1% vs 75.3%), escalation (94.7% vs 93.7%), and undisputed payment decision (89.0% vs 85.5%). The models present complementary profiles — classification strength vs arithmetic strength — within a shared structural ceiling.
Format constraints suppress reasoning on the hardest cases
Four cases in the hardest tier — including a dual-CSA netting construction — triggered prose-before-JSON output that exhausted the original 1,024-token budget before emitting parseable structure. At a 2,048-token budget with a format-tolerant salvage parser, all four cases resolved correctly. One case (dual-CSA netting) consistently required extended reasoning across all three runs even at the higher budget. Strict structured-output enforcement can silently penalize exactly the cases where model reasoning adds the most value.
Zero false positives: joint finding across both models
Claude Sonnet produced zero false positives across 378 clean-call runs, matching GPT-4o's result exactly. Combined across both models, 1,128 clean-call runs produced zero false positives, with a 1.0% upper CI bound for each. For operations teams evaluating LLM deployment in the margin workflow, false positive rate is often the threshold question — and both frontier models clear it.