SIMM / UMR Initial Margin Dispute Detection
(Figures from the v1.2 runs.) AAL-D-007 puts bilateral initial-margin reconciliation in front of the frontier field. Under UMR, two counterparties independently compute IM on the same netting set using ISDA SIMM; when the two numbers diverge beyond tolerance, someone has to decide whether a dispute exists and what caused it. The dataset is 250 netting sets — 162 genuine disputes across nine categories at 18 cases each (sensitivity discrepancies, CRIF mapping errors, FX conversion, concentration add-ons, calculation-date staleness, methodology version drift, trade-population gaps, bucket misassignment, netting-set scope) and 88 within-tolerance traps that look like disputes but are not. Version 1.2 corrects a defect in which one category's ground truth named evidence the case did not disclose; all nine models were re-run against the corrected dataset.
Nine models were evaluated — GPT-5.6 Sol, GPT-6 Astra, Claude Opus 5, Claude Sonnet 5, Gemini 3.1 Pro, Grok 4.5, DeepSeek V4 Pro, Kimi K3 and Meta's Muse Spark 1.2 — three runs each, deterministic scoring, no model permitted to compute. The finding inverts the usual shape. Detection is saturated: every model at 100.0%, a 0.0% false-flag rate on all 88 traps, and — uniquely in the corpus — a 0.0% run-to-run flip rate, meaning no model ever contradicted itself about whether a dispute exists. The wall sits on attribution.
Category accuracy runs 49.8% (DeepSeek V4 Pro) to 74.5% (Claude Opus 5). Underneath that sits a second finding: run each case three times and value answers move, with a flip rate spanning 3.1% (GPT-6 Astra) to 37.0% (DeepSeek V4 Pro). A benchmark run once cannot distinguish a reliably mediocre model from an unreliable one. And underneath both sits a third: the two categories the field fails almost completely are not arithmetic problems.
One asks a model to notice that a risk class the netting set declares in scope is populated on one side and absent on the other. The other asks it to notice a single risk factor carrying identical magnitude with an opposite sign, adjacent to factors that match exactly. Both are failed near-uniformly on a dataset where the computation-heavy categories score 100%.
Detection is not just solved — it is stable
(v1.2 runs.) All nine models post 100.0% detection accuracy with a 0.0% false-flag rate across all 88 within-tolerance traps. Escalation is 100.0% on eight of nine, with DeepSeek V4 Pro at 99.7%. More striking: across 250 cases and three runs each, no model ever gave a different verdict on whether a dispute exists. Accurate and self-consistent together is the strongest form of this finding anywhere in the corpus.
The wall sits on attribution
(v1.2 runs.) In D-004 through D-006 the ceiling sat on the dollar figure. Here value accuracy is comparatively strong (67.7–87.0%) while category attribution runs 49.8–74.5%. Models can tell you a dispute exists and how large it is; asked why, across nine equally-weighted categories, most of the field sits well short of reliable. Claude Opus 5 leads on category (74.5%) and on value (87.0%); DeepSeek V4 Pro trails on both (49.8% / 67.7%). Kimi K3 leads on component identification (82.3%) while sitting seventh on category (59.1%) — locating the offending component and naming why it is wrong are separable capabilities.
A single run cannot tell you what you have
(v1.2 runs.) Running each case three times exposes something a pooled accuracy figure hides. Value flip rate — the share of disputed cases where a model gets the IM amount right on some runs and wrong on others — spans 3.1% (GPT-6 Astra) to 37.0% (DeepSeek V4 Pro). Better than one in three disputed cases produces a different figure from DeepSeek depending on which run you read. Its 67.7% value accuracy is an average over a partly random process, not a stable capability, and only multi-run methodology surfaces the difference. All nine models are measured on the same 162 eligible disputed cases.
What the whole field cannot do is compare, not compute
(v1.2 runs.) Two of the nine categories are failed near-uniformly. IM-DIS-NETTING is answered correctly by none of the nine models across all three runs. IM-DIS-CRIF is answered correctly by none of the nine except Claude Sonnet 5, at 20%. Following the v1.2 correction, both were checked against the underlying case data before publication, using the test that identified the CALCDATE defect: does the case actually disclose the evidence its ground truth names? Both do. In the NETTING cases the netting set explicitly lists a risk class as in scope, the firm's sensitivities for that class are populated, and the counterparty's are an empty object. In the CRIF cases a single risk factor appears on both sides with identical magnitude and opposite sign, directly adjacent to factors matching to the decimal. Neither task is arithmetic. Both are structural comparisons — identify a missing section, identify a flipped sign — and both are failed by nine models from seven labs on the same dataset where concentration add-ons and methodology drift score 100% across the board. These models can compute. What they do not reliably do is hold two structured objects side by side and identify what differs between them. Middle- and back-office reconciliation is largely that task.
Escalation is a property of the task, not the model
(v1.2 runs.) Eight of nine models post 100.0% escalation accuracy here; DeepSeek V4 Pro posts 99.7%. The same DeepSeek posted 5.6% escalation on AAL-D-006 — an adjacent middle-office workflow — a 94-point swing on the same model. Escalation reliability cannot be inherited from a vendor scorecard or carried across deployments; it has to be measured on the workflow where the model will run.
v1.2 correction — what changed and why
On 5 Sep 2026 a per-category audit — accuracy broken out by root-cause category rather than pooled — found that IM-DIS-CALCDATE was answered correctly by none of the eight models then published, on any of three runs: zero of 486 observations. Independent models from different labs do not agree perfectly on a genuinely hard problem, so the cases were re-examined. The category describes one side's figure being computed against a stale calculation date, but the case disclosed a single shared calculation date and no staleness evidence of any kind; the stated root cause was unrecoverable from the material the model was given. Every model answered IM-DIS-SENS, which was the only reading the evidence supported. The models were right and the ground truth was unreachable. The generator has been fixed — each party's actual calculation date is now disclosed per side, and the within-tolerance trap carries differing dates too so that materiality still has to be judged. In v1.2, IM-DIS-CALCDATE is answered correctly by five of nine models at 100% and by every model above zero. The full roster, including GPT-6 Astra, was re-run against the corrected dataset and every figure on this page is from that run. The previously published category-attribution range of 45.7–63.4% was understated: roughly one ninth of its denominator was a question with no answer in it. AAL's own pre-publication gate did not catch this — it compared scorecards against this entry, both derived from the same scoring. A per-category degeneracy check now runs before the gate, and any category the whole field fails is checked against source data before it is published as a finding.
Comparability of the figures
Every model has all 750 runs scored, so every figure on this page rests on the same base: 486 disputed-case observations for attribution and value, and the same 162 eligible cases for flip rate. No run is excluded and none is imputed. An earlier version of this page, published before the failed runs were repeated, reported ranges on differing per-model denominators (attribution 47.2% to 73.5%, value flip 2.0% to 37.5%); those figures are superseded by the ones here.
v1.1 methodology correction
v1.0 always injected the dispute-causing perturbation into the counterparty's copy, which meant value accuracy could be earned by copying the firm's figure rather than by judgment. The flaw was caught on 19 Aug 2026; all v1.0 results — including a GPT-5.6 Sol run that scored 100/100/100 — were invalidated and archived rather than deleted. v1.1 balances the perturbation 125/125 across firm and counterparty sides and removes the prompt hint indicating which side is authoritative. v1.2 retains that balance; the gate confirms 125/125 on the corrected dataset.
Roster and known limitations
Nine models complete: GPT-5.6 Sol, GPT-6 Astra, Claude Opus 5, Claude Sonnet 5, Gemini 3.1 Pro, Grok 4.5, DeepSeek V4 Pro, Kimi K3, Muse Spark 1.2. A tenth, Alibaba's Qwen 3.8-Max, was attempted four times and never completed; it is excluded by name rather than reported, and appears in no figure on this page. Every model has all 750 runs scored, with none excluded. At first publication many runs had returned no valid answer, from provider credit limits, rate limiting and a few malformed outputs, and they were excluded rather than imputed. After provider credits were topped up, every failed run was repeated on 8 Oct 2026, and all figures on this page are from the completed data. One request setting differs between cells: the repeated runs for Grok 4.5, Kimi K3, DeepSeek V4 Pro and Muse Spark 1.2 requested reasoning explicitly through OpenRouter, while the cells that survived from the first run did not, so those four models mix two request settings (see the reasoning-mode section below). Correction, 7 Oct 2026: an earlier version of this page reported 22 errors for Grok 4.5, twelve for Kimi K3 and one for GPT-5.6 Sol, and said all other models recorded none. Those figures came from the earlier v1.1 results, not v1.2, and the statement that every model was within the gate's error-rate envelope was wrong. Correction, 8 Oct 2026: from 7 to 8 Oct this page listed per-model counts of runs with no valid answer (up to 348 of 750) and called the rankings provisional. Those failed runs have now been repeated, the counts no longer apply, and the ranges here replace the earlier ones. GPT-6 Astra rejects temperature, top_p and top_logprobs and therefore could not be run under the same sampling configuration as the other eight — that is disclosed here because it is a limitation on comparability, and is itself a finding about evaluating models whose sampling controls differ.
Model checkpoint, added Oct 2026
DeepSeek V4 Pro was requested on OpenRouter under the model name deepseek/deepseek-v4-pro. OpenRouter's model listing, read on 18 Oct 2026, maps that name to the checkpoint dated 23 April 2026 (deepseek-v4-pro-20260423). DeepSeek published a later Pro checkpoint on 13 Aug 2026 (deepseek-v4-pro-0813); it was not tested here. These runs did not record the checkpoint on each call, so this attribution rests on OpenRouter's listing rather than on our own logs. Runs from 18 Oct 2026 request dated checkpoints and record the served model.
AI assistance in construction
Cases and ground truth for this dataset were built by the author with AI assistance. Which model assisted was not recorded at the time, so we cannot say whether it belongs to a model family evaluated here. Where the project's commit records name an assistant, it is a Claude model, and Claude models are among those we evaluate. Scoring is deterministic code that no model influences. The exposure is in how the answers were built, not in how they are graded.
Build provenance — a disclosed gap in this run
DeepSeek V4 Pro was accessed through OpenRouter, which routes a single model name across many third-party hosts rather than serving it directly. This run recorded only the gateway ("openrouter") and not which upstream host answered each call, so the specific build that produced these results is not recoverable after the fact. DeepSeek separately announced on 10 Sep 2026 that calls to this model name would be served by a different model from 14 Sep, then reverted that in its own change log without dating the reversal; both statements remain live and contradictory. Nothing in the results indicates a wrong or inconsistent build was scored — the figures on this page are reported as run — but the provenance cannot be independently verified, and that is a limitation on reproducibility rather than a defect in the data. The evaluation driver has since been changed to record the upstream host and the build string the host reports back on every call. This gap is disclosed here for the same reason as the GPT-6 Astra sampling limitation below: a reproducibility claim that omits a known unverifiable is worth less than one that names it.
Build provenance, revisited: reasoning mode and flip rates
A follow-up test run after publication partly resolves the gap disclosed above. Hosts serving DeepSeek V4 Pro through OpenRouter default reasoning on or off independently of one another. In the AAL-D-008 pinned-host test (25 Sep 2026), on identical calculation cases, hosts defaulting reasoning off scored 5 to 7 percent while hosts defaulting it on scored 91 to 99 percent; with reasoning explicitly requested, the off-default hosts reasoned and answered correctly. The response does not reveal which mode served it. This run recorded reasoning tokens on every call, so its results can be split after the fact. 133 of DeepSeek V4 Pro's 750 calls, about 18 percent, returned no reasoning tokens, all of them in cells run before the driver was changed. When the failed runs were repeated on 8 Oct 2026 the driver requested reasoning explicitly, and all 194 repeated DeepSeek calls reasoned. Detection was 100 percent either way. Category attribution was 55 percent with reasoning and 28 percent without, and value accuracy was 73 percent with reasoning and 41 percent without, which lowers the published value figure by roughly six points. The larger effect is on consistency. Where all of a case's runs reasoned, DeepSeek's value flip rate was 30 percent (34 of 112 cases); where runs mixed reasoning on and off, it was 63 percent (24 of 38). Kimi K3, also served through OpenRouter, had 50 of 750 calls (7 percent) without reasoning, again all in the earlier cells, and its flip rate barely moved: 9.5 percent with reasoning throughout, 8.7 percent when mixed (2 of 23), so no host effect is visible for it. The published flip rate for DeepSeek V4 Pro therefore includes a component caused by host routing rather than model behaviour. With reasoning consistently on, the model-attributable figure is about 30 percent, against 37.0 published. DeepSeek remains the least stable model on value even with routing removed. Earlier public statements describing it as the least reliable model across two benchmarks should be read with this qualification; the other benchmark, AAL-D-006, recorded no reasoning telemetry and cannot be checked the same way. AAL's evaluation drivers now request reasoning explicitly on every aggregator call and record the serving host and reasoning tokens.
External validation — inference backend as a variance source
(v1.2 runs.) Two independent 2026 studies corroborate this page's category-attribution spread (49.8-74.5%) and value flip rate (3.1-37.0%) as reflecting a documented, structural issue in LLM evaluation generally, not a defect unique to this dataset. "What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend" (arXiv:2608.04714) found that inference backend choice alone, under greedy sampling-noise-free decoding, accounts for roughly 39% of the out-of-the-box score variance a practitioner sees, with the effect strongly model-dependent. Pape, Evertz & Schönherr, "The Silent Hyperparameter" (arXiv:2605.19537), independently found backend-alone discrepancies large enough to flip leaderboard rankings. Both papers recommend disclosing the inference backend, its version, and the full generation configuration. This dataset already discloses one such limitation above — GPT-6 Astra's rejection of temperature, top_p and top_logprobs — and going forward D-series runs will document backend and version per model alongside sampling configuration.