P&L Reconciliation Break Attribution
AAL-D-005 measures whether frontier models can adjudicate the daily P&L reconciliation every trading desk runs: front office reports a number, product control independently computes another, a risk explain accounts for part of the gap, and the model must decide whether the residual is a real break or a fully-explained move — then attribute the break to a root cause. The dataset is 250 cases (162 breaks across 15 BRK-* categories, 88 explained-move traps) evaluated across six models — Claude Sonnet 5, Claude Sonnet 4.6, GPT-4o, Gemini 2.5 Pro, DeepSeek V3.2, and KIMI K3 — three runs each, deterministic scoring, no model permitted to perform arithmetic. Detection is universally solved (94.8–98.9%). Value extraction — pinning the actual dollar figure — ranges from 55.4% down to 11.9%, replicating the quantification wall on a fifth workflow. The headline result: the three reasoning models span 26.9% to 55.4% on value, so 'reasoning model' does not predict performance; the specific architecture does.
Benchmark design: breaks vs. explained-move traps
Each case presents three views of a day's P&L — front-office estimate, product-control books, and a risk explain — and prints the unexplained residual and the materiality threshold. 162 cases are real breaks spanning 15 root causes (stale marks, missing or new trades, amendments, sign errors, FX, fees, corporate actions, day-count, sensitivities that do not reconcile, and more). 88 are explained-move traps: large P&L swings that look like breaks but are fully accounted for (a trade booked today, a dividend already applied, a residual below the materiality threshold, a known T+1 timing adjustment). The model reads, attributes, and recommends; it is forbidden from computing, so all arithmetic is generator-side and ground-truth by construction.
Detection is solved; quantification is the wall
Every model clears 94% on detection (GPT-4o 98.9%, both Sonnets 98.4%, Gemini 98.1%, KIMI K3 98.0%, DeepSeek 94.8%) — a fifth consecutive dataset where knowing a break exists is a solved language problem. Value extraction is the wall: 55.4% (Sonnet 5), 43.4% (KIMI K3), 39.4% (Sonnet 4.6), 30.3% (Gemini), 26.9% (DeepSeek), 11.9% (GPT-4o). Models reliably know a break happened, usually know its category and owner, and cannot reliably state the number.
The finding: 'reasoning model' does not predict performance
On AAL-D-004, thinking won the traps because those traps were operational conventions. D-005's traps are printed-number comparisons (is the residual below the threshold?), and that edge disappears — false-break rates collapse to 0–5% for nearly everyone (GPT-4o 0.0%, both Sonnets and KIMI ~0.4–4.5%), regardless of reasoning. More striking: the three reasoning models scatter across the value range. Sonnet 5 (thinking) tops the field at 55.4%; KIMI K3 (thinking) lands at 43.4%; DeepSeek V3.2 (thinking) trails at 26.9%, below a non-thinking Claude. Three reasoning models span 28 points — as wide as the whole field. The label predicts nothing; the architecture is everything. Notably, KIMI is the second-strongest model on the judgment layer (category 84.2%, field 79.8%, false-break 0.4%) yet still hits the same quantification wall — strong reasoning buys attribution, not arithmetic.
Methodology
Cases are generated deterministically; ground truth is established by independent Python repricing with a full audit trail (every reported residual reconciles under a separately-written recompute check before anything ships). Six models, three runs each at vendor-recommended sampling (reasoning models at their specified temperature/top-p, not forced to temperature 0), deterministic scoring on detection, false-break rate, explained-move accuracy, category attribution, field identification, value extraction, escalation, resolution action, and break owner. Scorer locked at d005-score-v1.0.1. A pre-publication gate independently re-runs the dataset recompute, the scorer contract tests, and per-model error/null checks; all six models passed at 250 cases × 3 runs with zero errors.
The deployment bottleneck
A model at 98% detection and 84% attribution but 43% value is a triage engine, not a booking engine. The configuration the data supports: route the judgment (is this a break, what kind, who owns it) to the LLM, route the arithmetic to deterministic code, and keep a human on the residual. And do not select models by the 'reasoning' label — select by measured task performance, because the label does not predict the result.
How to cite
BibTeX: @dataset{aal_d_005_2026, title={AAL-D-005: P&L Reconciliation Break Attribution Dataset}, author={{AI Alpha Labs}}, year={2026}, month={aug}, version={1.0}, publisher={AI Alpha Labs}, url={https://aialphalabs.ai/research/AAL-D-005}, note={250 cases, 162 breaks across 15 categories and 88 explained-move traps; six models; 4,500 scored observations}}. APA style: AI Alpha Labs. (2026). AAL-D-005: P&L Reconciliation Break Attribution Dataset (Version 1.0). Retrieved from https://aialphalabs.ai/research/AAL-D-005.