How AI Alpha Labs evaluates models
This page describes how AI Alpha Labs evaluates models today, including what it does not do.
Seven principles
These are the principles we work to. Where our practice falls short of one, the relevant results page says so. Two things this page does not describe, because they are not part of our process: independent human reviewers, and a human analyst baseline. Ground truth is fixed by the case generator, and the author reviews a sample of cases by hand with AI assistance. Each dataset page states what is known about which AI model assisted its construction. In most cases the model was not recorded, and because Claude models are among those we evaluate, we cannot rule out that a dataset was built with help from a model family it scores.
Evaluations are conducted independently of model providers. No provider reviews, approves, or influences methodology, scoring, or findings.
Methodology, scoring criteria, confidence levels, limitations, and failure modes are disclosed on each results page.
Sufficient information is published for an independent party to replicate the evaluation. A finding that cannot be reproduced is not a finding.
Findings are derived from data. Conclusions do not exceed what the evidence supports. Qualitative assessments are labeled as such.
Evaluations are conducted on tasks modeled on real institutional workflows. General-purpose or academic tasks are not used.
Scoring is performed by deterministic code that receives only the case and the output being scored, never the identity of the model.
Known limitations are documented before findings are drafted. Limitations are published with equal prominence to findings.
Nine-phase lifecycle
Every evaluation follows this sequence, and each phase produces defined artifacts.
Research Design
Research question · objectives · scope · workflow selection
Case Construction
Generators build the cases · correct answer fixed before any model sees them
Ground Truth Construction
Correct answer fixed before evaluation · author reviews a sample of cases for answerability
Pilot Run
Two-model pilot · categories every model fails checked against source data
Model Evaluation
Prompt lock · three runs per case · host, settings and reasoning recorded
Deterministic Scoring
Code scores each output against the answer key · no reviewer, no model judge
Statistical Analysis
Metrics · Wilson 95% intervals computed by case (3 runs grouped) · run-to-run flip rate
Limitations and Threats to Validity
Internal · external · construct · documented before findings
Publication
Findings · version · release · corrections published where they apply
Terms and definitions
Defined terms are used consistently across all publications.
The proportion of tasks where the model's output matches ground truth on all required fields, as determined by deterministic scoring code.
An evaluation procedure in which the scorer does not know which model produced the output being scored. In AAL evaluations the scorer is code that receives only the case and the output.
A quantitative measure of the statistical robustness of an evaluation's findings. Confidence reflects the quality of the evidence, not the performance of any model.
Any output element not present in, or directly derivable from, the input provided to the model. Synonymous with 'hallucination' in colloquial usage.
The correct output for a given task, fixed before model evaluation begins.
A model that meets all evaluation thresholds simultaneously for a given workflow.
A predefined set of criteria that determines how model outputs are scored against ground truth. Rubrics are fixed before evaluation begins.
A category of operational work evaluated under this methodology (e.g., trade confirmation matching, reconciliation break analysis).
Materials
Generators, scorers, evaluation drivers, datasets and results are in the public repository. Each research page states the models, run settings and known limitations for that benchmark.
Want to audit our methodology? Every dataset, prompt, and scorer is in the public repository for independent replication.
View the repository on GitHub →