The Reproducibility Crisis in Capital Markets AI Benchmarking
The capital markets technology sector faces a reproducibility crisis in AI benchmarking: model versions are vague, dataset composition is undisclosed, evaluation rubrics are proprietary, and results rest on a single run of a single model. This note contrasts that norm with the AI Alpha Labs protocol — a fully documented, synthetic-by-construction, cross-model, multi-run framework spanning 1,000 cases across four datasets (trade confirmations, margin disputes, equity-options exceptions, settlement fails), built for roughly $150–200 in total API cost. We identify three validation gaps that survive even when a dataset is disclosed — fine-tuning leakage, circularity, and stochastic underestimation — and illustrate them with a contamination incident from our own corpus that our audit gate caught before publication. We conclude with a due-diligence checklist and a vendor Reproducibility Scorecard, and argue that reproducibility is the scarce resource in capital markets AI deployment.
Reproducibility, not accuracy, is the scarce resource
When benchmarks cannot be reproduced, competition shifts from evidence to marketing — the vendor with the best deck wins, not the vendor with the most reliable system. A CRO weighing a $500K contract has no basis for judging whether an accuracy claim is robust, cherry-picked, or fabricated. If rigorous multi-model evidence can be produced for ~$150–200, as the AAL corpus demonstrates, the absence of such evidence from vendor claims is a choice, not a constraint.
Synthetic construction closes the leakage gap
AAL datasets are generated deterministically from locked specifications rather than drawn from client production documents. This is the foundation of the corpus's reproducibility: there is no real document set a model could have memorized, no client-confidentiality constraint on disclosure, and the full dataset, prompts, and deterministic scorer can be published after evaluation without redaction. A pre-publication SHA-256 registration and a no-leakage grep on serialized prompts close the residual paths.
A checklist and scorecard for procurement
The note ends with a 10-question due-diligence checklist and a vendor Reproducibility Scorecard designed to be sent as a pre-qualification questionnaire — covering model version, dataset composition, run count, cross-model validation, published rubric, data provenance, leakage control, arithmetic-validation architecture, marginal cost, and independent replication. AI Alpha Labs completed the scorecard on its own corpus (9.5/10, 0.5 deducted for pending independent replication) to demonstrate the standard.
The 10-question Reproducibility Scorecard
Send this to any AI vendor before accuracy claims are discussed. Each question scores Yes = 1, Partial = 0.5, No = 0 (10 points total); a vendor scoring below 5 cannot substantiate its claims. (1) Model version disclosure — the exact model version(s) used. (2) Dataset size & composition — total cases and stratification by asset class, workflow, and difficulty. (3) Independent runs — runs per case and the sampling protocol (temperature). (4) Cross-model validation — were multiple frontier models run on the same cases, and how were disagreements handled? (5) Published evaluation rubric — field definitions, scoring rules, edge-case handling. (6) Data provenance — production, synthetic, or vendor-created; if production, separation from training data. (7) Leakage control — confirm no test case appeared in any training or tuning set. (8) Arithmetic-validation architecture — how numerical claims are validated; does the LLM compute, or does code? (9) Marginal cost disclosure — API cost to reproduce one evaluation pass. (10) Independent replication — has a third party replicated it, or what is the external-replication policy? Print-ready scorecard (with the AAL self-assessment): https://www.aialphalabs.ai/AAL-Reproducibility-Scorecard.html