GPT-4o on AAL-D-002
The first complete evaluation on AAL-D-002, the margin call dispute benchmark. GPT-4o was run three times across all 250 cases and scored by the deterministic engine across seven metrics. Detection and classification performance is strong: 99.9% detection, 94.9% category accuracy, 0.0% false positive rate. The critical finding is amount accuracy: 76.1% — confirmed in a workflow where computing the correct margin call amount is the explicit primary task. The arithmetic weakness is not a trade-confirmation artifact — it is a structural limitation in financial calculation that persists across fundamentally different workflows.
Protocol
Three independent runs per case at temperature 0 using the OpenAI API (gpt-4o-2024-08-06). All 250 cases from AAL-D-002-v1.0 were evaluated in a single pass with a resume-safe checkpoint system. One API error occurred across 750 calls (0.13% error rate). All accuracy figures carry Wilson 95% confidence intervals computed over pooled run-case observations.
Detection and classification
GPT-4o correctly identifies whether a dispute exists in 99.9% of cases — one miss across 750 observations. False positive rate is 0.0% with a 1.0% upper CI bound. Category accuracy at 94.9% is meaningfully higher than on AAL-D-001 (90.2%), likely because dispute categories in AAL-D-002 are more mutually exclusive and better defined than trade confirmation exception types.
Amount accuracy: the critical finding
Amount accuracy sits at 76.1%. AAL-D-002 was designed so that arithmetic is the primary deliverable: every dispute case requires applying haircuts, threshold deductions, rounding conventions, and netting arithmetic to produce a correct dollar figure. Despite this, the model computes the correct amount in only 3 of 4 cases. Systematic failures appear on threshold-adjusted calculations, haircut math, and SIMM dispute amounts.
Cross-dataset comparison
GPT-4o exposure accuracy on AAL-D-001 was 68.2%. Amount accuracy on AAL-D-002 is 76.1% — an 8-point improvement attributable to simpler integer arithmetic in dispute amounts versus FX rate pip calculations. The structural finding is unchanged: detection (~99.9%) is 24 points above amount accuracy (~76%) — confirmed across two independent datasets and two fundamentally different capital markets workflows.
Undisputed payment and escalation
Undisputed payment accuracy (89.0%) measures whether the model correctly determines what to pay while a dispute is pending — operationally critical for avoiding settlement risk. Escalation accuracy (94.7%) reflects correct routing of severe disputes (DIS-DIR, DIS-ELGBLTY) while handling below-MTA threshold notifications and clean calls as non-escalation events.