Settlement Fail Root Cause Identification — Equity Options
AAL-D-004 applies the trap-clean methodology to settlement failure root-cause identification on equity options. Where AAL-D-003 showed detection saturation (100% across all models), D-004 engineers trap cases that defeat every non-thinking model identically (100% FP on three trap families) while partially defeated by Sonnet 5's reasoning (27–50% residual FP depending on trap type). The benchmark reveals that reasoning solves computation-shaped traps (netting, permitted-partial settlements) but not convention-shaped ones (late-but-completed settlements). Four models — Claude Sonnet 5, Claude Sonnet 4.6, GPT-4o, and Gemini 2.5 Pro — were evaluated 750 times each (3 runs × 250 cases) with deterministic scoring and full consensus auditing.
Benchmark design: trap cleans
AAL-D-004 uses three families of engineered trap cases — clean settlements that are not live exceptions. Late-but-settled traps: a settlement that completed after the exception-detection window closes (no remediation needed). Permitted-partial traps: a partial settlement that the CSA permits without escalation (reconciled against future deliveries). Netting traps: a settlement that was netted into another contract (correctly excluded from isolated case inspection). Non-thinking models treat all three as exceptions (100% FP). Sonnet 5 (the only reasoning model) solves netting entirely (0% FP), mostly solves permitted-partial (27% FP), but remains fooled by late-but-settled (89% FP). This trap design is durable — reasoning depth does not rescue convention-shaped problems.
Thinking vs. computation: where reasoning helps and doesn't
Sonnet 5 pays for its detection advantage with a categorization regression on hard cases: FAIL-NET drops from 95% (Sonnet 4.6) to 48%, FAIL-ACCT from 100% to 10%. The thinking model allocates reasoning budget differently, sacrificing categorization precision for gate accuracy. Reasoning solves the computational traps (netting arithmetic is irreducible computation; Sonnet 5 nails it) and partially solves allowed-settlement logic (a partially-computable rule). It does not solve the late-but-settled problem because that requires knowing an operational convention — settlements are no longer live exceptions once they land in the accounting system — not deeper computation.
Methodology
Cases are generated deterministically, ground truth via independent Python repricing with full audit trail (all 250 cases independently repriced by a separately-written script before anything ships). Three runs per case at temperature 0, with deterministic scoring on detection, categorization (EXC-PRICE, EXC-QTY, EXC-SETTLEMENT, EXC-NETTING, EXC-ACCT, etc.), field identification, exposure (±$1,000), and operational judgment (should_escalate, remediation_action). All scored observations stored; consensus auditing on the 3 runs per case before aggregation. Every failure is documented.
The deployment bottleneck
No model is deployable end-to-end on this workflow without a human on remediation. The strongest configuration the data supports is a division of labour: (1) a thinking model for the fail/no-fail gate (Sonnet 5 at 15.2% FPR), (2) a cross-vendor ensemble for categorization (~95% on exceptions via majority vote), and (3) deterministic playbook code — not an LLM — for remediation choice once the category is known. This mirrors real operations: model outputs inform human judgment; they do not replace it.
Sample case preview
Case AAL-D-004-042 · Equity Option · Permitted-Partial Settlement Trap. Single-name put, 200 shares, strike $95, expiry 3 months. Confirmed settlement: 50 shares (partial, allowed by CSA schedule). Internal record shows all 200 shares unsettled. Ground truth: no exception (settlement is permitted, future deliveries reconcile the balance). Scoring: detection (is this a live exception?) must be False; if model says True, it fails as a false positive. Sonnet 5 correctly identifies this as permitted (no exception) in 2/3 runs; Sonnet 4.6, GPT-4o, and Gemini treat it as an exception 3/3 times (false positive).
AI assistance in construction
Cases and ground truth for this dataset were built by the author with AI assistance. Which model assisted was not recorded at the time, so we cannot say whether it belongs to a model family evaluated here. Where the project's commit records name an assistant, it is a Claude model, and Claude models are among those we evaluate. Scoring is deterministic code that no model influences. The exposure is in how the answers were built, not in how they are graded.
How to cite
BibTeX: @dataset{aal_d_004_2026, title={AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset}, author={{AI Alpha Labs}}, year={2026}, month={jul}, version={1.0}, publisher={AI Alpha Labs}, url={https://aialphalabs.ai/research/AAL-D-004}, note={250 cases with trap cleans discriminating thinking from non-thinking models; 3000 scored observations}}. APA style: AI Alpha Labs. (2026). AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset (Version 1.0). Retrieved from https://aialphalabs.ai/research/AAL-D-004. MLA style: AI Alpha Labs. AAL-D-004: Settlement Fail Root Cause Identification—Equity Options Dataset, version 1.0, July 2026, aialphalabs.ai/research/AAL-D-004.