AI ALPHA LABS · INDEPENDENT BENCHMARKS
Can frontier models be trusted with trade-ops work?
We run every case three times and publish accuracy and run-to-run consistency. Every number on this page is computed from the result files in our public repository; each chart links to its source folder.
Same model, different host
DeepSeek V4 Pro on the same 97 AAL-D-008 cases, pinned to each serving host: share of answers with the correct value.
Figure 1. Value accuracy by serving host, DeepSeek V4 Pro, AAL-D-008 v0.1, as of 2026-09-29.
Errored responses count as incorrect (Azure 3, DeepInfra 0, DigitalOcean 3, Novita 0). Reasoning on or off follows each host's default.
Source: eval_out_d008/pinned_host_deepseek.json · AAL-D-008 is v0.1
Source: eval_out_d008/pinned_host_deepseek.json · AAL-D-008 is v0.1
Accuracy isn't the whole story
AAL-D-007, v1.2 runs (SIMM/UMR initial margin disputes): value accuracy vs. how often a model's value answer changed across its own 3 runs.
- 1claude-opus-5 (87.0%, 3.7%)
- 2gpt-5.6-sol (85.8%, 7.4%)
- 3moonshotai/kimi-k3 (85.2%, 9.3%)
- 4meta/muse-spark-1.2 (83.3%, 16.7%)
- 5gemini-3.1-pro-preview (82.5%, 4.3%)
- 6x-ai/grok-4.5 (79.4%, 16.0%)
- 7claude-sonnet-5 (76.5%, 21.6%)
- 8gpt-6-astra (68.9%, 3.1%)
- 9deepseek/deepseek-v4-pro (67.7%, 37.0%)
Each numbered dot is one model, numbered by value accuracy (highest first); the key gives (value accuracy, value flip). Where two markers would overlap, one is moved and a line points to its true position. All markers are one colour; colour carries no meaning.
Figure 2. Value accuracy vs. run-to-run value flip rate, AAL-D-007 v1.2 (v1.2 runs), as of 2026-10-12.
Shown for all 9 models in the published v1.2 flip-rate file. Each model's flip rate is counted over the same 162 disputed cases. Hover a point for its values.
Source: eval_out_d007_v12 (scorecard_d007_*.json, flip_rate_d007_v12.json), 250 cases × 3 runs. As of 2026-10-12 (last commit to this folder).
Source: eval_out_d007_v12 (scorecard_d007_*.json, flip_rate_d007_v12.json), 250 cases × 3 runs. As of 2026-10-12 (last commit to this folder).
Leaderboard: AAL-D-008 Repo Trade Disputes (v0.1)
250 cases, three runs each. Click a column to sort. AAL-D-008 Repo Trade Disputes (v0.1): full results page →
| Model | Detection | Attribution | Value ↓ | False positive rate | Attribution flip rate |
|---|---|---|---|---|---|
| claude-opus-5-5 | 100.0% | 100.0% | 100.0% | 0.0% | 0.0% |
| x-ai/grok-4.7 | 100.0% | 100.0% | 98.2% | 0.0% | 0.0% |
| gpt-5.6-sol | 100.0% | 94.5% | 93.1% | 0.0% | 4.8% |
Table 1. AAL-D-008 v0.1 leaderboard, as of 2026-09-29.
As of 2026-09-29 (last commit to this folder) · Source: eval_out_d008 (scorecard_d008_*.json)
AAL-D-007 value accuracy, all 9 models (v1.2 runs)
| Model | Value accuracy | Value flip | Errored runs | File |
|---|---|---|---|---|
| claude-opus-5 | 87.0% (n=486) | 3.7% (6/162) | 0 | scorecard_d007_claude-opus-5.json |
| gpt-5.6-sol | 85.8% (n=486) | 7.4% (12/162) | 0 | scorecard_d007_gpt-5-6-sol.json |
| moonshotai/kimi-k3 | 85.2% (n=486) | 9.3% (15/162) | 0 | scorecard_d007_moonshotai_kimi-k3.json |
| meta/muse-spark-1.2 | 83.3% (n=486) | 16.7% (27/162) | 0 | scorecard_d007_meta_muse-spark-1-2.json |
| gemini-3.1-pro-preview | 82.5% (n=486) | 4.3% (7/162) | 0 | scorecard_d007_gemini-3-1-pro-preview.json |
| x-ai/grok-4.5 | 79.4% (n=486) | 16.0% (26/162) | 0 | scorecard_d007_x-ai_grok-4-5.json |
| claude-sonnet-5 | 76.5% (n=486) | 21.6% (35/162) | 0 | scorecard_d007_claude-sonnet-5.json |
| gpt-6-astra | 68.9% (n=486) | 3.1% (5/162) | 0 | scorecard_d007_gpt-6-astra.json |
| deepseek/deepseek-v4-pro | 67.7% (n=486) | 37.0% (60/162) | 0 | scorecard_d007_deepseek_deepseek-v4-pro.json |
Table 2. AAL-D-007 v1.2 (v1.2 runs) value accuracy and value flip rate by model, as of 2026-10-12.
Source: eval_out_d007_v12 (scorecard_d007_*.json, flip_rate_d007_v12.json)