Settlement Fail Root Cause Identification — Equity Options
AAL-D-004 applies the trap-clean methodology to settlement failure root-cause identification on equity options. Where AAL-D-003 showed detection saturation (100% across all models), D-004 engineers trap cases that defeat every non-thinking model identically (100% FP on three trap families) while partially defeated by Sonnet 5's reasoning (27–50% residual FP depending on trap type). The benchmark reveals that reasoning solves computation-shaped traps (netting, permitted-partial settlements) but not convention-shaped ones (late-but-completed settlements). Four models — Claude Sonnet 5, Claude Sonnet 4.6, GPT-4o, and Gemini 2.5 Pro — were evaluated 750 times each (3 runs × 250 cases) with deterministic scoring and full consensus auditing.
Benchmark design: trap cleans
AAL-D-004 uses three families of engineered trap cases — clean settlements that are not live exceptions. Late-but-settled traps: a settlement that completed after the exception-detection window closes (no remediation needed). Permitted-partial traps: a partial settlement that the CSA permits without escalation (reconciled against future deliveries). Netting traps: a settlement that was netted into another contract (correctly excluded from isolated case inspection). Non-thinking models treat all three as exceptions (100% FP). Sonnet 5 (the only reasoning model) solves netting entirely (0% FP), mostly solves permitted-partial (27% FP), but remains fooled by late-but-settled (89% FP). This trap design is durable — reasoning depth does not rescue convention-shaped problems.
Thinking vs. computation: where reasoning helps and doesn't
Sonnet 5 pays for its detection advantage with a categorization regression on hard cases: FAIL-NET drops from 95% (Sonnet 4.6) to 48%, FAIL-ACCT from 100% to 10%. The thinking model allocates reasoning budget differently, sacrificing categorization precision for gate accuracy. Reasoning solves the computational traps (netting arithmetic is irreducible computation; Sonnet 5 nails it) and partially solves allowed-settlement logic (a partially-computable rule). It does not solve the late-but-settled problem because that requires knowing an operational convention — settlements are no longer live exceptions once they land in the accounting system — not deeper computation.
Methodology
Cases are generated deterministically, ground truth via independent Python repricing with full audit trail (all 250 cases independently repriced by a separately-written script before anything ships). Three runs per case at temperature 0, with deterministic scoring on detection, categorization (EXC-PRICE, EXC-QTY, EXC-SETTLEMENT, EXC-NETTING, EXC-ACCT, etc.), field identification, exposure (±$1,000), and operational judgment (should_escalate, remediation_action). All scored observations stored; consensus auditing on the 3 runs per case before aggregation. Every failure is documented.
The deployment bottleneck
No model is deployable end-to-end on this workflow without a human on remediation. The strongest configuration the data supports is a division of labour: (1) a thinking model for the fail/no-fail gate (Sonnet 5 at 15.2% FPR), (2) a cross-vendor ensemble for categorization (~95% on exceptions via majority vote), and (3) deterministic playbook code — not an LLM — for remediation choice once the category is known. This mirrors real operations: model outputs inform human judgment; they do not replace it.
Sample case preview
Case AAL-D-004-042 · Equity Option · Permitted-Partial Settlement Trap. Single-name put, 200 shares, strike $95, expiry 3 months. Confirmed settlement: 50 shares (partial, allowed by CSA schedule). Internal record shows all 200 shares unsettled. Ground truth: no exception (settlement is permitted, future deliveries reconcile the balance). Scoring: detection (is this a live exception?) must be False; if model says True, it fails as a false positive. Sonnet 5 correctly identifies this as permitted (no exception) in 2/3 runs; Sonnet 4.6, GPT-4o, and Gemini treat it as an exception 3/3 times (false positive).
How to cite
BibTeX: @dataset{aal_d_004_2026, title={AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset}, author={{AI Alpha Labs}}, year={2026}, month={jul}, version={1.0}, publisher={AI Alpha Labs}, url={https://aialphalabs.ai/research/AAL-D-004}, note={250 cases with trap cleans discriminating thinking from non-thinking models; 3000 scored observations}}. APA style: AI Alpha Labs. (2026). AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset (Version 1.0). Retrieved from https://aialphalabs.ai/research/AAL-D-004. MLA style: AI Alpha Labs. AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset, version 1.0, July 2026, aialphalabs.ai/research/AAL-D-004.