LLM Benchmark Contamination: How a Quota Error Corrupted 29% of Our Gemini Eval
Before publishing the Gemini 2.5 Pro results from our margin call dispute benchmark, we ran audit_evals.py — our cross-model eval integrity checker — across every result file. We found 218 contaminated runs across 78 cases. We didn't publish them. Instead, we purged the bad data and we're re-running clean.
This is what went wrong, how we caught it, and why having an audit gate matters more than having a perfect pipeline.
The Contamination
We were evaluating Gemini 2.5 Pro against AAL-D-002, our 250-case margin call dispute benchmark. The harness was configured for three runs per case (750 total), 1-second sleep between calls.
Midway through the eval — around case 78 of 250 — the Gemini API returned a quota exhaustion error. The harness caught the exception. Instead of failing loudly and stopping, it logged the error as a prediction and moved on. The prediction object contained dispute_exists: null — not a model answer, just a failed API call recorded as a scored result.
The resume logic then did what it was supposed to do: on restart, it skipped cases that had completed their 3 runs. But for the 78 cases that hit quota, "completed" meant "has an error recorded." The contaminated runs were never retried.
Result: 218 of 750 runs (29%) across 78 of 250 cases (31%) were API failures recorded as model errors.
How We Caught It
We have a read-only audit script, audit_evals.py, that inspects every result file for five failure signatures: swallowed API errors, parse errors, null detections, truncation suspects, and salvaged outputs.
Running it across the Gemini D-002 results flagged 218 runs across 78 cases — all null dispute_exists. That's the telltale signature of an API error being recorded as a prediction.
Final verdict: CONTAMINATED.
What We Did
We wrote a purge script that removes contaminated runs from the results file, backing up the original first. Cases with zero clean runs are dropped entirely; the harness's resume logic then treats them as fresh when you run it again.
After purging, we re-ran those 78 cases with a clean API quota. The new scorecard:
Model: Gemini 2.5 Pro | Cases: 250 | Runs: 3 | Errors: 0 Detection accuracy: 65.6% (CI [62.1%, 68.9%]) Category accuracy: 60.5% (CI [55.4%, 65.3%]) Amount accuracy: 44.5% (CI [39.4%, 49.8%]) Escalation accuracy: 66.1% (CI [61.2%, 70.8%]) False positive rate: 9.0% (CI [6.5%, 12.3%])
Errors: 0. No API failures. No null predictions. The data is clean. This is Gemini's actual performance on margin call dispute matching.
Why This Matters
If we had published without auditing, detection accuracy would have been suppressed by ~29%. Gemini would have ranked last in a three-way comparison it might actually win. The error would look like a model weakness, not a pipeline failure.
Frontier model evals are tight. Pipeline noise dominates the signal. Audit gates are not optional; they are the difference between a benchmark and a rumor.
What We're Doing Next
Publish clean three-model results (AAL-R-2026-042-v1.1) with GPT-4o, Claude Sonnet, and Gemini 2.5 Pro all audit-confirmed. Open-source the audit tooling. Bake audit_evals.py into the standard eval gate: no results ship without passing it.
If you run LLM benchmarks of any kind, you have pipeline failure modes. The question is whether you're checking for them or just hoping they don't exist.
We're checking.