Reasoning Buys Computation, Not Convention: What Settlement-Fail Traps Reveal

By the third benchmark, detection stopped being interesting. Every frontier model catches every real exception — trade confirmations, margin disputes, equity-options breaks, all at or near 100%. If detection is solved, a benchmark that only measures detection measures nothing. So for AAL-D-004 we stopped asking whether models can find the problem and started asking whether they can resist finding problems that aren't there.

The answer separates the field by more than 30 points — and it tells you exactly what reasoning does and doesn't buy.

The benchmark: settlement fails and the traps that aren't fails

AAL-D-004 is 250 settlement failure cases in the equity-options lifecycle. 162 carry a real exception; the other 88 are trap cleans — settlements that look broken but require no action. Three trap families do the work.

Late-but-settled: a settlement that completed after the exception-detection window closed. It failed, then cured. No remediation needed. Permitted-partial: a partial settlement the CSA explicitly allows, reconciled against future deliveries. Netting: a settlement netted into another contract, correctly excluded from isolated inspection.

Every trap is a clean case dressed as a failure. Ground truth comes from an independent Python repricing script — written from the spec, sharing no code with the generator — that re-prices all 250 cases before anything ships. Four models: Claude Sonnet 5 (the only reasoning model in the set), Claude Sonnet 4.6, GPT-4o, and Gemini 2.5 Pro. Three runs per case, 3,000 scored observations.

Detection saturates. The false-positive rate is the whole story.

All four models catch 100% of the 162 real exceptions. Not one live failure slips through, on any run, for any model. That number is now table stakes.

The discriminator is what happens on the 88 traps — the false-positive rate, the rate at which a model cries wolf on a settlement that's actually fine.

Claude Sonnet 5 (thinking) — 15.2% · Claude Sonnet 4.6 — 38.6% · GPT-4o — 38.6% · Gemini 2.5 Pro — 49.6%

The two non-thinking Claude and OpenAI models land in an identical spot. Gemini is worse. And the reasoning model is more than twice as clean as the field. In a real ops queue, that gap is the difference between an alert stream a human can triage and one they learn to ignore.

Where reasoning actually helps

Sonnet 5's advantage is not uniform — and the shape of it is the finding. Broken out by trap family:

Netting traps: 0% false positives. Solved completely. Permitted-partial traps: 27% false positives. Mostly solved. Late-but-settled traps: 89% false positives. Barely moved.

The non-thinking models fail all three the same way — roughly 100% false positives across every trap family, because they treat any settlement that looks wrong as an exception.

So reasoning doesn't buy a general immunity to false alarms. It buys the ability to solve traps that are computational. Netting is arithmetic: does this settlement reconcile against the contract it was netted into? That's an irreducible calculation, and the thinking model runs it. Permitted-partial is partly computable — check the settlement against a CSA rule. Sonnet 5 gets most of the way there.

Where it doesn't

Late-but-settled is not a computation. It's a convention. A settlement that failed and then cured before the accounting system closed the window is no longer a live exception — not because any number says so, but because that's how operations treats it. There's nothing to calculate. You either know the convention or you don't.

Sonnet 5 doesn't. It stays fooled 89% of the time, right alongside the non-thinking models. More reasoning depth does not rescue a convention-shaped problem, because reasoning has nothing to chew on. This is the sharpest version of a pattern we've now seen four datasets running: the frontier is separating on computation, and standing still on convention.

The cost of thinking: a categorization regression

Sonnet 5 doesn't get its gate accuracy for free. On the cases where it must not just detect but categorize the failure, it regresses hard against its non-thinking sibling. FAIL-NET categorization drops from 95% (Sonnet 4.6) to 48%. FAIL-ACCT drops from 100% to 10%.

The thinking model spends its reasoning budget on the gate decision — is this a real exception? — and takes it out of the categorization decision — what kind? Sonnet 4.6, which agonizes less about whether to flag, ends up the most accurate categorizer on exceptions in the whole field at 86.1%.

That's a genuine tradeoff, not a bug. If your workflow needs a clean gate, the thinking model wins decisively. If it needs precise routing of a confirmed break, a non-thinking model categorizes better. You don't get both from one model.

What this means if you run a settlement desk

The deployment that falls out of this isn't 'use the best model.' It's an assembly. Use the thinking model as the gate — it's the only one that keeps false alarms low enough to trust. Use an ensemble or a non-thinking model for categorization, where reasoning depth actively hurts. And put the remediation itself in a deterministic playbook, because the part that decides what to do about a confirmed netting fail is a rule, not a judgment.

The broader lesson is a buying signal. When a vendor tells you their model 'reasons about' your exception workflow, ask which half of the workflow. Reasoning is real, measurable capability on the computational traps. On the ones that come down to how your desk conventionally treats a cured fail, it buys you nothing — and no amount of model upgrade will, until the convention is written into the system as a rule.

AAL-D-004 methodology, per-trap-family breakdowns, and the full consensus audit are documented in our research index. The dataset and ground truth stay private to keep future evaluations uncontaminated.