A pre-qualification questionnaire for evaluating AI vendor benchmark claims in capital markets operations.
Most AI accuracy figures cited in capital markets procurement are neither reproducible nor independently validated. Send this scorecard to any AI vendor before accuracy claims are discussed. A vendor declining to complete it should be disqualified from further consideration.
| # | Question | Vendor Response | Score |
|---|---|---|---|
| 1 | Model version disclosureState the exact model version(s) used (e.g., gpt-4o-2024-08-06, claude-sonnet-5). List all if multiple. | — | ☐ /1 |
| 2 | Dataset size & compositionTotal cases evaluated, and stratification by asset class, workflow, and difficulty. | — | ☐ /1 |
| 3 | Independent runsRuns per case and the sampling protocol (temperature and any variation). | — | ☐ /1 |
| 4 | Cross-model validationWere multiple frontier models run against the same cases? List them; describe disagreement handling. | — | ☐ /1 |
| 5 | Published evaluation rubricAttach or link the full rubric: field definitions, scoring rules, edge-case handling. | — | ☐ /1 |
| 6 | Data provenanceProduction, synthetic, or vendor-created? If production, confirm separation from training data. | — | ☐ /1 |
| 7 | Leakage controlConfirm no test case appeared in any training or tuning set; describe the separation or construction protocol. | — | ☐ /1 |
| 8 | Arithmetic-validation architectureHow are numerical claims validated? If the LLM does arithmetic, how is it verified? If code does, describe the handoff. | — | ☐ /1 |
| 9 | Marginal cost disclosureApproximate API cost to reproduce one evaluation pass at current pricing, with model(s) and tokens per case. | — | ☐ /1 |
| 10 | Independent replicationHas a third party replicated the benchmark? If yes: name, date, findings. If no: state your external-replication policy. | — | ☐ /1 |
| TOTAL | /10 |
We hold ourselves to the standard we ask of vendors. Scored against our own benchmark corpus: 2,000 cases across 8 datasets (AAL-D-001 through D-008), with results published for 3 (D-006 to D-008). As of Oct 12, 2026.
| # | Criterion | AAL Response | Score |
|---|---|---|---|
| 1 | Model version disclosure | Exact model strings are recorded in every published scorecard. Published scorecards cover 13 models: 10 on D-006, 9 on D-007, 3 on D-008 (counted from the scorecard_*.json files in the aal-benchmark eval_out_d006, eval_out_d007 and eval_out_d008 folders). | 1 |
| 2 | Dataset size & composition | 2,000 cases across 8 datasets, 250 each. D-001: 250 (7 asset classes). D-002: 250 (17 categories). D-003: 250 (19 categories). D-004: 250 (162 fails / 88 trap cleans). D-005: 250 (162 breaks / 88 explained-move traps). D-006: 250 (162 ineligible / 88 traps, 12 eligibility categories). D-007 v1.1: 250 (162 disputes / 88 traps, 9 categories). D-008 v0.1: 250 (227 discrepancies / 23 no-dispute, 11 root-cause labels). Stratified by difficulty. | 1 |
| 3 | Independent runs | Three runs per case. Temperature 0 where the endpoint allows it; reasoning models use vendor-recommended settings, recorded per run. Where shown, Wilson 95% intervals are computed by case (the 3 runs of each case grouped). | 1 |
| 4 | Cross-model validation | Multiple models on identical cases, three runs each: 7,500 scored observations on D-006 (10 models), 6,750 on D-007 (9 models), 2,250 on D-008 (3 models), per the published scorecards. Disagreements surfaced. | 1 |
| 5 | Published evaluation rubric | Deterministic, criteria-keyed scorer with per-case scoring_criteria; full dataset specifications published. | 1 |
| 6 | Data provenance | Synthetic, ground-truth-by-construction from locked specifications — not client production data. Fully disclosable; datasets, prompts, scorers published post-evaluation. | 1 |
| 7 | Leakage control | No leakage path exists — cases derive from no real corpus. Pre-publication SHA-256 registration; no-leakage grep confirms no ground-truth string in any serialized prompt. | 1 |
| 8 | Arithmetic-validation architecture | Deterministic Python performs all arithmetic; LLMs classify only. No model performs a day-count, exposure, call amount, or residual. Non-negotiable across all datasets. | 1 |
| 9 | Marginal cost disclosure | As of AAL-RS-009 (Jul 2026): ~$150–200 for the first 750 cases (5 models, 3 datasets). A current cost for the full 2,000-case corpus is not yet published. | 0.5 |
| 10 | Independent replication | Not yet independently replicated. Specifications, prompts, and scripts published to enable it; treated as the outstanding step. | 0.5 |
| AAL TOTAL | 0.5 deducted for pending independent replication; 0.5 deducted because the only published cost figure covers the first 750 cases, not the full corpus. | 9.0 / 10 |
Complete and return with any vendor proposal. Incomplete scorecards should be returned for revision; vendors scoring below 5 should not advance to subsequent evaluation rounds. Verify claims by (1) requesting the prompt templates, (2) running a 50-case subset through the vendor's stated protocol and disclosed model versions, and (3) comparing the vendor's reported accuracy against your independent run. A claim that cannot survive this verification should not survive contract negotiation.