AI Alpha Labs
BENCHMARKS · AAL-D-008 · v0.1

AAL-D-008: Repo Trade Disputes (v0.1)

Given both counterparties' records of a repo trade and the governing confirmation, can a model detect whether a dispute exists, name its root cause, and state the corrected figure?

Updated 2026-09-29 (last commit to eval_out_d008) · 250 cases · 3 runs per case · 3 models

As of 2026-09-29, claude-opus-5-5 ranks first on D-008 with 100.0% value accuracy, followed by x-ai/grok-4.7 (98.2%) and gpt-5.6-sol (93.1%).

Value accuracy is scored on the 199 cases with a rate or amount to correct, 3 runs each (597 runs). The other 51 cases (28 insufficient-data, 23 no-dispute) have no value to score; detection and attribution are scored on all 750 runs. [3]

Leaderboard

RankModelValue acc.95% CIAttributionDetectionFalse pos.Attr. flipLatency p50 / p95 (s)Errors
1claude-opus-5-5100.0%98.1–100.0100.0%100.0%0.0% (n=69)0.0%4.9 / 9.60
2x-ai/grok-4.798.2%95.2–99.3100.0%100.0%0.0% (n=69)0.0%13.3 / 93.90
3gpt-5.6-sol93.1%88.7–95.994.5%100.0%0.0% (n=69)4.8%3.4 / 7.70
Table 1. AAL-D-008 v0.1 leaderboard, ranked by value accuracy (ties broken by attribution), as of 2026-09-29.
Value accuracy is scored on the 199 cases with a rate or amount to correct, 3 runs each (597 runs). The other 51 cases (28 insufficient-data, 23 no-dispute) have no value to score; detection and attribution are scored on all 750 runs. [3] 95% intervals are Wilson intervals computed over the 199 value-scored cases (each case's 3 runs counted as one observation), by this page's build script from the eval_results files. Latency is per call as recorded by the driver and depends on the serving host. Cost per case is not shown: the result files record tokens but no prices.
Scores: eval_out_d008 (scorecard_d008_*.json). 95% CI: eval_out_d008 (eval_results_d008_*.json, recomputed by case). [4]

Key Takeaways

  • All 3 models detected whether a dispute existed in every one of their 750 observations, and none flagged a dispute on the 23 no-dispute trap cases (69 observations each).
  • The models separate on the corrected figure: value accuracy runs from 100.0% (claude-opus-5-5) to 93.1% (gpt-5.6-sol).
  • gpt-5.6-sol's attribution misses are concentrated in one category: RL-DIS-RECALL at 57.3%, against 100.0% in every other category. It changed its root cause between runs on 12 of 250 cases.
Derived from per_category_attribution and attribution_flip_rate in the scorecards [4].

Background

Each case gives a model two counterparties' records of the same repo trade and a governing record (the confirmation, plus any recall or corporate action notice). All 250 cases in v0.1 are repo trades [1]. Cases are generated deterministically with no LLM calls, and each correct answer is computed by the generator before any model sees the case [2]. Scoring is plain Python comparison against that ground truth; no model grades another model [3].

Cases model real operations workflows but are not drawn from live trade records, and several generator parameters are assumptions [5]. Some ranges in the generator are explicitly marked as unsourced placeholders pending domain-expert review [2], and 0 of 250 cases are flagged as manually verified against a source [1]. The case identifier shown to models includes the benchmark name; whether this affects behaviour was not tested [5].

Results

1. claude-opus-5-5100.0% (95% CI 98.1–100.0)
2. x-ai/grok-4.798.2% (95% CI 95.2–99.3)
3. gpt-5.6-sol93.1% (95% CI 88.7–95.9)
80%85%90%95%100%
Value accuracy; axis starts at 80%. Bracket = 95% Wilson interval.
Figure 1. Value accuracy with 95% Wilson interval by model, AAL-D-008 v0.1, as of 2026-09-29.
95% intervals are Wilson intervals computed over the 199 value-scored cases (each case's 3 runs counted as one observation), by this page's build script from the eval_results files.
Source: eval_out_d008 (eval_results_d008_*.json, recomputed by case) [4]

Worked Example: AAL-D-008-0003 (RL-DIS-RECALL)

Counterparty A records a notional of $31,414,745.77 and accrued interest of $331,661.18; counterparty B records $49,486,448.31 and $522,453.18. Both show a rate of 4.223%, ACT/360, trade date 2026-06-29, settlement 2026-09-27. The governing record gives an original notional of $49,486,448.31 and a recall notice of $18,071,702.54.

Correct answer (dataset ground truth [1]): root cause RL-DIS-RECALL, corrected amount $331,661.18.

Deterministic computation (same formula as the generator [2], re-run by this page's build script):

outstanding = original_notional − recall_notice
            = 49,486,448.31 − 18,071,702.54 = 31,414,745.77
days        = 2026-09-27 − 2026-06-29 = 90
accrual     = outstanding × 0.04223 × 90 / 360 = 331,661.18

gpt-5.6-sol answered this case incorrectly on 2 of 3 runs. Run 1 output:

root_cause RL-DIS-NOTIONAL, corrected_amount $49,486,448.31: “The notionals differ by more than $1,000. The authoritative confirmation supports Counterparty B's notional of $49,486,448.31; the accrued-interest difference results from the notional mismatch.”

claude-opus-5-5 answered RL-DIS-RECALL, $331,661.18 on 3 of 3 runs. Run 1 output:

“Both sides agree on rate (4.223%), ACT/360 and dates (2026-06-29 to 2026-09-27, 90 days). The notionals differ by 49,486,448.31 - 31,414,745.77 = 18,071,702.54, which is exactly the confirmation's recall notice amount. The confirmation fixes the original notional at 49,486,448.31 and also records a recall of 18,071,702.54, so the outstanding balance is 31,414,745.77. A's accrued interest of 331,661.18 (31,414,745.77 x 4.223% x 90/360) reflects the recall. B accrued 522,453.18 on the full pre-recall notional, so the error is on B's side.”
Sources: datasets/AAL-D-008-v0.1.json, eval_results_d008_gpt-5-6-sol.json, eval_results_d008_claude-opus-5-5.json

Methodology & Dataset

Each model ran all 250 cases 3 times (750 observations). A failed call is recorded as an error, not as a wrong answer [6]. Each response is scored for detection (is there a dispute), attribution (root-cause category) and value (corrected amount or rate) [3]. Dataset generator seed: 8 [1].

ModelProviderPromptSampling
claude-opus-5-5anthropicd008-prompt-v0.1provider_default
x-ai/grok-4.7openrouterd008-prompt-v0.1temperature=1.0
gpt-5.6-solopenaid008-prompt-v0.1temperature=provider_default [endpoint rejected: temperature]
Root-cause categoryCases
RL-DIS-RECALL32
RL-DIS-INSUFFICIENT-DATA28
RL-DIS-NOTIONAL24
RL-DIS-HAIRCUT23
RL-DIS-DAYCOUNT23
RL-DIS-CORPACTION23
RL-NO-DISPUTE23
RL-DIS-ACCRUAL20
RL-DIS-SETTLEDATE19
RL-DIS-BENCHRATE19
RL-DIS-FEESPLIT16
Table 2. AAL-D-008 v0.1 cases by root-cause category (total 250; 23 flagged as traps).
Source: datasets/AAL-D-008-v0.1.json (ground_truth.correct_root_cause, case_input.is_trap). Code and data: github.com/yuschajp/aal-benchmark (code MIT, data CC BY 4.0).

Changelog

v0.1 (2026-09-29): first published release, 250 cases, 3 models. See eval_out_d008 and the repository history.

How to cite

AI Alpha Labs. AAL-D-008: Repo Trade Disputes (v0.1). 250 cases. Updated 2026-09-29. https://www.aialphalabs.ai/benchmarks/d-008

References

  1. AAL-D-008 v0.1 dataset (250 cases, ground truth)
  2. AAL-D-008 case generator, generate_d008.py
  3. AAL-D-008 deterministic scorer, score_d008.py
  4. AAL-D-008 scorecards and raw model outputs, eval_out_d008/
  5. aal-benchmark README, including known limitations
  6. Baseline driver, run_d008_baseline.py

Want this kind of evaluation on your own workflow? See the $4,500 entry audit, or email us about an audit.