Independent AI evaluation for capital markets operations.

AI Alpha Labs was founded to bring independent, reproducible benchmarking to a market where every AI vendor publishes its own accuracy numbers and nobody audits the methodology.

Capital markets operations is a consequential domain. A missed margin call dispute, a misbooked trade confirmation, a settlement failure that wasn't flagged — these are not abstract errors. The AI being evaluated for these workflows deserves evaluation by someone with no stake in the result.

AAL builds benchmark datasets from real capital markets operations workflows — trade confirmation matching, margin call dispute resolution, settlement exception identification. Each dataset contains hundreds of cases with pre-constructed ground truth, scored by a deterministic engine rather than model judgment.

We run frontier AI models from OpenAI, Anthropic, Google, xAI, Meta, DeepSeek, Moonshot and Alibaba against these datasets and publish the results with full methodology. Every evaluation uses multiple runs, Wilson 95% intervals computed by case (the 3 runs of each case grouped) where intervals are shown, and a contamination audit gate before any result is published.

The research is free and public. The commercial products — AAL Certified for vendors, Buy-side Audits for firms — apply the same methodology to external systems under contract.

Detection is solved. Reliability is not.

Across 2,000 benchmark cases and eight datasets, one pattern holds: frontier models now detect operational exceptions at or near 100%. What varies is everything after detection. On earlier datasets, models computed the correct dollar figure between 12% and 76% of the time. On the newest, frontier models with reasoning enabled computed every figure correctly, while the same model scored 5% on identical cases when its host served it with reasoning switched off. Models also change their answers between identical runs, and fail when a case leaves a convention unstated. Reliability depends on the model, its configuration and how precisely the task is specified, and none of that shows up in a single accuracy number.

This finding drives the architecture of the AAL API: language models handle classification and extraction; a deterministic Python engine handles all arithmetic. The benchmark is the evidence for the architecture. See the published results →

Regulators are asking the same question.

IOSCO's AI toolkit for market participants lists the absence of independent validation of model performance as a named supervisory concern, and directs supervisors to ask firms using outside AI vendors whether those systems have been tested using the firm's own data — and to ask vendors for the evidence behind public benchmark claims, not just the claims themselves.

The toolkit is non-binding, and "independent" in its usage most likely means independent of a model's own developers inside a firm, not a third-party evaluator specifically. AAL does not claim IOSCO endorsement. The overlap is worth noting because it describes, from a standard-setting body rather than a vendor, the exact gap this research exists to close: benchmark claims without the evidence behind them.

Measured against a published standard.

On September 18, 2026, the AI Evaluator Forum published Minimum Conditions for Embedding Evaluators, signed by more than 100 researchers and evaluators. It sets three conditions an evaluator must meet to be considered independent of the systems it assesses. AI Alpha Labs is not a signatory and makes no claim of endorsement. We hold ourselves to the same conditions and report our position against each.

No ownership by a model developer. AI Alpha Labs is independently owned and self-funded. No AI lab or model developer holds any equity or control in the company.

No significant commercial business with a model developer. We purchase API access from the labs whose models we evaluate, at standard published rates, as any customer does. We receive no payment, credits, discounts, early access or preferential treatment from any of them. Our commercial relationships are with firms and vendors who commission benchmarks and audits of systems deployed in capital markets operations, not with the developers of the underlying models.

No payment contingent on findings. Commissioned work is priced at a fixed fee agreed before any evaluation runs. The fee does not change with the result. Clients pay for the measurement, not the verdict.

Who pays for what. A commissioned engagement is paid for by the party whose system is being tested, which is why its fee is fixed before any work begins. Published research is different: no benchmark in our public registry was funded by any developer, vendor or firm whose models or systems it evaluates. We keep the two separate so a reader can always tell which is which.

Where our methodology has failed, we have said so on the relevant results page rather than amending figures quietly. Those corrections are part of the record.

How we work.

Eight datasets. 2,000 cases. All public.