Every benchmark, evaluation and finding, published in full.
The complete AI Alpha Labs corpus. Each entry carries its status, its evidence level, and a link to the materials that make it reproducible. Nothing is published until it is defensible.
2,000 cases across 8 datasets, with results published for 3. See the leaderboard and charts.
Repo and Securities Lending Rate Disputes
250 repo and securities lending cases: 199 disputes across nine categories, 23 within-tolerance traps and 28 cases built to be unanswerable. Frontier models with reasoning on saturate the benchmark. GPT-5.6 Sol and Claude Opus 5.5 score 100% on detection, attribution and value with zero run-to-run variation, whether computed figures are printed or withheld. The findings sit around that number: every failure found during construction was a defect in the test, and one model scored 5% or 99% on identical cases depending on which host served it.
SIMM / UMR Initial Margin Dispute Detection
(v1.2 runs.) 250 bilateral initial-margin reconciliation cases — 162 disputes across 9 IM-DIS categories, 88 within-tolerance traps — under ISDA SIMM, across nine frontier models. Detection is not merely solved but stable: 100.0% on every model, 0.0% false flags, and zero run-to-run contradictions across all 250 cases. Naming the cause of the dispute runs 49.8–74.5%, and run-to-run value instability spans 3.1% to 37.0%. Two categories are failed near-uniformly by the entire field, and both are structural comparisons rather than computations.
A 30B open-weight model ties frontier detection on AAL-D-006
Meta's Muse Glimmer (30B, open-weight, single-GPU local deployment) posts 95.7% detection on AAL-D-006, level with Claude Opus 5, and 78.6% substitution validity, ahead of two frontier models. Its value accuracy of 16.3% sits at the low end of the frontier range on this dataset (15.4% to 57.8%). On D-006, model scale did not predict value accuracy.
Collateral Eligibility & Substitution
250 collateral cases — 162 ineligible across 12 eligibility categories, 88 traps — across nine frontier models. Detection is solved (93–100%); no model values the exposure above 58%; escalation splits the field from 5.6% to 98.8%. A companion cut tests whether a 30B open-weight model closes the gap — it doesn't.
P&L Reconciliation Break Attribution
250 daily P&L reconciliation cases — 162 breaks across 15 root-cause categories, 88 explained-move traps — across six frontier models including DeepSeek V3.2 and Kimi K3. Results pending a results file (2026-10-12): no D-005 results file has been found, so the model results on this page have been removed until it is located.
Arithmetic ceiling on AAL-D-002: second-model confirmation withdrawn
GPT-4o reaches 76.1% on correct call amounts against 99.9% detection on AAL-D-002. The second-model confirmation (Claude Sonnet 4.6, AAL-RS-008) is withdrawn pending its source results file.
Claude Sonnet 4.6 on AAL-D-002
Results withdrawn Oct 12, 2026, pending the source results file.
Financial arithmetic weakness is structural, not workflow-specific
Amount accuracy at 76.1% on AAL-D-002 — a workflow where correct calculation is the primary deliverable — confirms that the arithmetic limitation first observed in AAL-D-001 is structural, not an artifact of the trade confirmation task.
Dual-exception recall and exposure accuracy vary meaningfully across model families
GPT-4o and Gemini models achieve 100% dual-exception recall; Claude Sonnet achieves 87.4%. Exposure accuracy: Gemini 2.5 Pro 76.0%, Gemini 2.5 Flash 79.5%, GPT-4o 66.1%, Claude 64.3% (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases).
Benchmark specification gaps can masquerade as model capability differences
Gemini 2.5 Pro's escalation accuracy rose from 75.7% in v1.0 to 99.0% in the later eval_out_v2 run after adding one paragraph of policy guidance — a case study in how underspecified benchmarks produce misleading model scores.
Models detect exceptions far more reliably than they quantify them
Detection sits at ~99.5% while exposure accuracy on the later run is 76.0% for Pro and 79.5% for Flash (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases).
Residual errors concentrate in judgment-heavy booking exceptions
Three independent frontier models miss the same judgment-heavy derivative cases while clearing 99%+ of everything else — residual risk lives in entity- and structure-level reasoning.
Detection accuracy clusters near ceiling on objective exceptions
Across four independent model evaluations across three labs, first-pass detection on objective exceptions sits at 97.9–99.6% — the pattern is now confirmed across models, architectures, and labs.
The Reproducibility Crisis in Capital Markets AI Benchmarking
Most AI accuracy figures cited in capital markets procurement are neither reproducible nor independently validated. This note contrasts that norm with the AAL protocol — at publication, 1,000 synthetic-by-construction cases across four datasets (now 2,000 cases across 8 datasets, with results published for 3), up to four frontier models at three runs each, all arithmetic in deterministic Python, built for ~$150–200 — and provides a vendor due-diligence checklist and a 10-question Reproducibility Scorecard for procurement teams. Reproducibility, not accuracy, is the scarce resource.
The Economics of LLM Deployment in Capital Markets Operations
The total API cost to build the AAL benchmark corpus — 750 cases, 5 frontier models, 3 datasets at the time of writing (since grown to 1,000 cases across four datasets) — was approximately $150–200. Commercial products built on that evidence base operate at 98–99% gross margins. The margin profile is structurally determined by an architecture that routes classification to LLMs and arithmetic to deterministic Python.
GPT-4o on AAL-D-002
99.9% detection, 94.9% category accuracy, 0.0% false positive rate across 750 observations — and amount accuracy of 76.1%, confirming the arithmetic weakness from AAL-D-001 survives into a workflow where correct calculation is the primary deliverable.
GPT-4o on AAL-D-001 (v1.2 multi-run)
99.2% detection, 100% dual-exception recall, 0 errors across 750 observations — closing the comparison table with full sub-metrics and confirming GPT-4o matches Gemini on all dimensions except exposure arithmetic.
Claude Sonnet 4.6 on AAL-D-001
98.8% detection, 0.0% false positive rate, 0 errors across 750 observations — with a lower exposure accuracy (64.3% on 57 cases (7 removed) (rescored 2026-10-12 on the corrected key; for the later runs, the prompt in our run scripts contained the same unit error, and these runs' answers are consistent with it, so 7 cases removed, 57 cases)) and dual-exception recall (87.4%) than Gemini, and a novel EXC-STAT false positive pattern requiring a v1.2 prompt fix.
Prompt Sensitivity Analysis: v1.0 vs v1.1 on AAL-D-001
Gemini 2.5 Pro's escalation accuracy rises from 75.7% to 99.0% with one paragraph of policy guidance. Exposure figures were re-scored after an answer-key error was found. Benchmark iteration separates specification gaps from genuine model limitations.
Gemini 2.5 Flash on AAL-D-001
99.5% detection (746/750), 100% dual-exception recall, zero errors across 750 observations. Corrected Oct 12, 2026: figures now come from the full v2 run.
Gemini 2.5 Pro on AAL-D-001
99.6% detection, 100% dual-exception recall, 0 errors across 750 scored observations — and a persistent exposure arithmetic weakness that survives prompt improvement.
Settlement Fail Root Cause Identification — Equity Options
250 equity-options settlement failure cases with trap cleans that separate thinking from non-thinking models — the first benchmark where false-positive rate, not detection, discriminates the field.
Equity-Options Confirmation Exceptions
250 equity-options confirmation cases with greeks, multi-leg spreads, and dual exceptions — plus a controlled prompt ablation across four frontier models.
Margin Call Dispute Detection — Dataset
250 validated cases covering margin call dispute detection, classification, and correct-amount calculation across OTC derivatives, exchange-cleared products, and securities lending.
Trade Confirmation Exception Identification
A production benchmark for whether frontier models can detect, classify, and quantify settlement exceptions across seven asset classes.
GPT-4o on AAL-D-001
First published evaluation: 97.9% first-pass detection across all 250 cases, with the single failure case documented rather than hidden.
Trade Confirmation Exception Identification — Dataset
250 validated cases pairing counterparty confirmations against internal records across cash and derivative products, with ground truth and per-case scoring criteria.