The Scarce Resource in Capital Markets AI Isn't Accuracy. It's Reproducibility.

A chief risk officer weighing a $500K AI contract is usually handed one number: an accuracy figure on a slide. What model version produced it? Which cases? How many runs? Who scored it, and against what rubric? On most vendor decks, none of those questions have answers. The number is not reproducible, and a number you can't reproduce is not evidence — it's marketing with a decimal point.

That's the actual state of AI benchmarking in capital markets. Not that the models are bad — that the claims about them can't be checked. We think reproducibility, not accuracy, is the scarce resource, and we can show why with our own corpus.

The math that makes the crisis a choice

Across four datasets — trade confirmations, margin disputes, equity-options exceptions, and settlement fails — the AAL benchmark corpus is 1,000 cases, each evaluated across up to four frontier models at three runs apiece, temperature 0, every accuracy figure carrying a Wilson 95% confidence interval. Zero percent of the arithmetic runs through the language model; all of it runs in deterministic Python.

The total API cost to build that evidence base was roughly $150 to $200.

That number is the whole argument. If a fully documented, cross-model, multi-run benchmark costs less than a nice dinner, then a vendor who publishes a single-run accuracy figure with no dataset and no rubric isn't constrained by cost. They're making a choice about what to disclose. Reproducibility this cheap turns its absence into a signal.

Disclosure isn't enough: three gaps that survive it

Even a vendor who publishes their dataset can still hand you a number you can't trust, because three validation gaps survive disclosure.

Fine-tuning leakage: if the model was tuned on data resembling the test set, a disclosed benchmark measures memorization, not capability. Circularity: if the same team wrote the cases, the ground truth, and the scorer, the benchmark can quietly encode its own answer key. Stochastic underestimation: a single run of a non-deterministic model is a sample of one — the reported number could be a lucky or unlucky draw, and you can't tell which without multiple runs and a confidence interval.

These are the reasons 'we published our dataset' is necessary but not sufficient. You have to know how the dataset was made, who validated it, and how many times the model actually ran.

How synthetic construction closes the leakage gap

AAL datasets aren't drawn from client production documents — they're generated deterministically from locked specifications. That single design decision does a lot of work. There's no real document set a model could have memorized, so leakage has nothing to leak. There's no client-confidentiality constraint, so the full dataset, prompts, and scorer can be published after evaluation without redaction. Ground truth comes from an independently written repricing script that shares no code with the generator, which breaks the circularity. A pre-publication SHA-256 registration and a no-leakage grep over the serialized prompts close the residual paths.

It's the difference between 'trust our number' and 'here is everything you'd need to reproduce it, and a hash proving we didn't change it after the fact.'

We caught ourselves — which is the point

Rigor isn't a claim you make; it's a gate that occasionally fails you in public. On our margin-dispute dataset, an API quota error silently recorded 29% of a run as model predictions instead of failures. Our audit gate caught it before publication. We purged the contaminated runs, re-ran clean, and wrote up exactly what happened.

Had we shipped it, Gemini would have looked ~29 points worse than it actually is — a pipeline failure disguised as a model weakness. A benchmark without an audit gate doesn't just risk that outcome; it can't even see it. The willingness to publish your own contamination incident is itself a reproducibility signal.

The 10-question scorecard to send any vendor

The research note ends with a due-diligence checklist you can drop straight into a pre-qualification questionnaire. Ten questions: exact model version and date; dataset composition and size; number of runs per case; cross-model validation; published scoring rubric; data provenance; leakage controls; whether arithmetic runs in the model or in code; the marginal cost to reproduce; and whether anyone independent has replicated it.

To demonstrate the standard rather than just assert it, we scored our own corpus against the checklist: 9.5 out of 10, with the half-point deducted because independent third-party replication is still pending. The scorecard is deliberately answerable by any honest vendor in an afternoon. The ones who can't answer it are telling you something.

What to do with this if you're buying

Stop grading vendors on the headline accuracy number and start grading them on whether the number can be checked. Send the scorecard before the demo, not after. Treat a refusal to specify model version, run count, or scoring method as a finding in itself. And ask the one question that collapses most of the ambiguity: does a dollar figure in your output come from the language model or from deterministic code? If it comes from the model, the number is plausible-looking by construction and wrong at a rate you can't predict.

Reproducibility is cheap to provide and expensive to fake. That asymmetry is the buyer's best tool. Use it.

The full note — AAL-RS-010, including the complete checklist and the Reproducibility Scorecard — is published in our research index.