Collateral Eligibility & Substitution
AAL-D-006 tests whether frontier models can adjudicate collateral eligibility and substitution: read a collateral schedule against an eligibility rulebook, decide what's ineligible, attribute the reason, value the exposure, propose a valid substitute, and decide whether to escalate. 250 cases (162 ineligible across 12 categories, 88 look-ineligible traps) across nine models — GPT-5.6 Sol, Claude Opus 5, Claude Sonnet 5, Gemini 3.1 Pro, Grok 4.5, DeepSeek V4 Pro, Kimi K3, Meta's Muse Spark 1.2, and Alibaba's Qwen 3.8-Max — three runs each, deterministic scoring, no model permitted to compute. Detection is universally solved (93.1–100%); value extraction tops out at 57.8% (Gemini 3.1 Pro) and bottoms at 15.4% (Grok 4.5). Two sharper findings: escalation accuracy spans 5.6% to 98.8% — wider than the value gap — and no single model wins across the board (Gemini 3.1 Pro leads on value, Opus 5 on escalation, Grok 4.5 on substitution — yet each hits the value wall). A separate cut (AAL-F-008) tests Meta's open-weight Muse Glimmer 30B against the same set: its detection falls within the frontier range, and its value accuracy (16.3%) sits at the low end of it.
On this dataset, scale did not predict value accuracy.
Detection is solved; quantification is the wall
Every model clears 93% detection; none exceeds 58% on the actual dollar figure. A sixth consecutive workflow where knowing an exception exists is a solved language problem and computing its value is not.
Escalation, not value, splits the field
Escalation accuracy runs from 5.6% (DeepSeek V4 Pro) to 98.8% (Claude Opus 5) — a wider spread than value, and the metric that most directly governs production trust. A model that flags the ineligible piece but can't reliably decide when to escalate is a liability, not a control.
Ninth flagship, same wall
Meta's Muse Spark 1.2 and Alibaba's Qwen 3.8-Max join the roster at 100% detection each. Spark 1.2 posts the field's second-best substitution validity (92.6%) and still pins value accuracy at 31.1%. Qwen 3.8-Max does better — 39.2% value accuracy, second in the entire corpus behind only Gemini 3.1 Pro — but is also the first model in the series with per-case telemetry attached, and it shows why: p95 latency of 311.6 seconds per case, driven by an average of 5,784 hidden reasoning tokens against roughly 73 tokens of actual answer. Two more frontier models, zero movement on the value ceiling — and the first hard evidence that accuracy and latency trade off independently of it.
Methodology
Ground truth by construction; every verdict, category, and exception value independently reconciled by a separate Python recompute before publication. Nine models, three runs each, deterministic scoring on detection, false-break rate, category attribution, field identification, value extraction, substitution validity, and escalation. A pre-publication gate re-runs the recompute, the scorer contract tests, and per-model error checks; all nine passed at 250 cases × 3 runs, eight with zero errors and Qwen 3.8-Max with 12 (well within the gate's error-rate envelope).
Model checkpoint, added Oct 2026
DeepSeek V4 Pro was requested on OpenRouter under the model name deepseek/deepseek-v4-pro. OpenRouter's model listing, read on 18 Oct 2026, maps that name to the checkpoint dated 23 April 2026 (deepseek-v4-pro-20260423). DeepSeek published a later Pro checkpoint on 13 Aug 2026 (deepseek-v4-pro-0813); it was not tested here. These runs did not record the checkpoint on each call, so this attribution rests on OpenRouter's listing rather than on our own logs. Runs from 18 Oct 2026 request dated checkpoints and record the served model.
AI assistance in construction
Cases and ground truth for this dataset were built by the author with AI assistance. Which model assisted was not recorded at the time, so we cannot say whether it belongs to a model family evaluated here. Where the project's commit records name an assistant, it is a Claude model, and Claude models are among those we evaluate. Scoring is deterministic code that no model influences. The exposure is in how the answers were built, not in how they are graded.
How to cite
BibTeX: @dataset{aal_d_006_2026, title={AAL-D-006: Collateral Eligibility & Substitution Dataset}, author={{AI Alpha Labs}}, year={2026}, month={aug}, version={1.1}, publisher={AI Alpha Labs}, url={https://aialphalabs.ai/research/AAL-D-006}, note={250 cases, 162 ineligible across 12 eligibility categories and 88 traps; nine models; 6,750 scored observations}}. APA style: AI Alpha Labs. (2026). AAL-D-006: Collateral Eligibility & Substitution Dataset (Version 1.1). Retrieved from https://aialphalabs.ai/research/AAL-D-006.
Run-to-run flip rate — added Sept 2026
D-006 was originally published with pooled accuracy only. Computing run-to-run flip rate against the existing three-run data surfaces a finding invisible in the original results: Claude Sonnet 5, which does not stand out as a weak performer in pooled accuracy, has a 34.6% value flip rate (56 of 162 eligible cases), meaning it gets the identical case right on some runs and wrong on others more often than DeepSeek V4 Pro (29.0%) or Qwen 3.8-Max (31.1%) — two models that score visibly lower on average. DeepSeek V4 Pro separately posts this dataset's highest detection flip rate (8.0%, 20 of 250 cases), consistent with its AAL-D-007 performance: the same model is the least self-consistent performer across two independent AAL datasets. Pooled accuracy alone did not surface either finding.