We Told Four Frontier Models Exactly What to Do. Half the Failure Vanished. The Other Half Is Arithmetic.

Our first benchmark ended on an unresolved note: every prompt fix we tried moved every metric except one. Exposure arithmetic — the dollar impact of a break — stayed broken. We couldn't tell how much of that was our prompt and how much was the models.

AAL-D-003 was built to answer that question properly: a harder dataset, and a controlled experiment. Same 250 cases, same deterministic scorer, four frontier models, two prompt versions — the only variable is what we tell the model.

The benchmark

AAL-D-003 is 250 equity-options confirmation-matching cases: single names and index products across Bloomberg-confirmed OTC, CME, and Eurex, with multi-leg spreads, dividend schedules, greeks, and corporate-action histories. 162 cases carry an injected exception across 19 categories; 88 are clean traps designed to invite false positives — post-split strikes that look wrong but aren't, deep in-the-money deltas near 1.0, negative theta.

Every numeric ground truth comes from a deterministic pricing engine, and an independent checker — written from the specification, sharing no code with the generator — re-prices every case. An adversarial AI review pass caught two defect classes the mechanical gates missed. We also audited our own scorer after the first run and found two bugs that were distorting model comparisons; we fixed them, rescored everything from stored predictions, and archived the before/after. The full audit trail ships with the results.

Models: Claude Sonnet 5, Claude Sonnet 4.6, Gemini 2.5 Pro, GPT-4o — three runs per case, 6,000 scored observations across both prompt versions.

Detection is solved. That's not the story.

All four models detected exceptions perfectly: 3,000 out of 3,000 decisions, including every clean trap, with zero false positives. Two years ago that would have been the headline. Today it means detection no longer discriminates between frontier models — the interesting failures live downstream, in classification and quantification.

The experiment: one prompt change

Version 1.0 of our prompt told models to report exposure and 'use null when the break has no direct dollar impact.' Version 1.1 changed two things: it stated that term breaks — settlement type, exercise style, expiry, currency, corporate actions — DO carry dollar impact and must be re-valued, and it defined category as root cause, not symptom.

That's the whole intervention. Same cases, same scorer, same everything else.

Half the failure was ours

Under v1.0, no model exceeded 48.8% on exposure. The dominant failure wasn't wrong arithmetic — it was models answering null on breaks they'd correctly detected, reasoning that a settlement-type mismatch has no direct dollar impact. Our prompt invited exactly that reading.

Under v1.1, exposure roughly doubled everywhere: Sonnet 4.6 from 34.1% to 63.0%, Gemini 2.5 Pro from 39.4% to 59.9%, GPT-4o from 21.7% to 39.1%, Sonnet 5 from 48.8% to 62.8%. Null answers collapsed — Gemini went from nulling 44% of exception cases to 0%.

Roughly half of what looked like an arithmetic failure was a specification failure. Any published benchmark number that doesn't test prompt sensitivity is measuring the prompt as much as the model.

The half that isn't

The other half survived. With a fully specified prompt, the best three models still get exposure wrong 37–40% of the time. The errors are not conceptual: contract-multiplier slips, per-contract versus total confusion, re-valuation mistakes. The models know what to compute. They compute it wrong.

This is the third dataset in a row — after trade confirmations broadly and margin call disputes — where the same pattern holds: extraction is solved, instructions recover recognition, arithmetic does not follow.

Same instruction, four ceilings

The taxonomy fix produced the sharpest capability separation we've measured. Corporate-action breaks manifest as a multiplier or quantity mismatch; the root cause is a missed split adjustment. Under v1.0, every model labeled the symptom — 0% across the board. Told explicitly to label root cause, the field spread out: Sonnet 5 reached 70%, Gemini 23%, Sonnet 4.6 13%, GPT-4o stayed at 0%.

An instruction can only unlock reasoning that's already there. Connecting a multiplier symptom to a corporate action it never sees stated is the hardest step in this benchmark, and it cleanly ranks the current frontier.

The side-effect we're fixing next

Sharpening the exposure rule had a cost: two of four models started equating 'no dollar impact' with 'no escalation needed.' GPT-4o stopped escalating counterparty-name breaks entirely — 12/12 correct before, 0/12 after. Sonnet 5 dropped similarly on greek breaks. Gemini improved and Sonnet 4.6 held flat, so the effect is model-specific — but it's a one-line fix for prompt v1.2: escalation follows risk criticality, not dollar size.

Every prompt change moves more than the metric you aimed at. Measure all of them.

What this means if you run an ops desk

First: prompt specification is free capability. If your team is evaluating models on operational workflows with underspecified prompts, your numbers are pessimistic by up to 30 points, and vendor comparisons at different specification levels are meaningless.

Second: the arithmetic gap is real, persistent, and silent. A model that nulls or miscomputes exposure produces output that looks exactly like correct output. If dollar amounts drive decisions, the calculation belongs in deterministic code — the model extracts, the engine computes. That's the architecture behind our discrepancy API, and after three benchmarks it's the only one we'd put in production.

AAL-D-003 methodology, scorecards, and the full audit trail are documented in our research index. The dataset and ground truth stay private — publishing them would contaminate every future evaluation run against it.