LLM Arithmetic Failures in Capital Markets: How We Caught and Fixed the Anti-Pattern

We spend a lot of time telling people not to let language models do arithmetic. It's the core finding behind our benchmark work: across every model and workflow we've tested, LLM arithmetic accuracy in capital markets tasks hits a structural ceiling somewhere between 68% and 85%. Extraction is a language problem. Reconciliation math is not. Our standing architectural rule — written into our project docs from day one — is that the LLM never touches arithmetic. All calculations run in deterministic Python.

Last week, an automated architecture review of our own dispute analysis endpoint flagged a violation. The rule was in the docs. It wasn't in the code.

What we found

Our margin call dispute endpoint (/v1/disputes/analyze) classifies incoming dispute notices against a firm's internal calculation — the workflow behind our AAL-D-002 benchmark of 250 dispute cases across 17 categories. The classification part is exactly what LLMs are good at: reading a free-text CSA notice, identifying the dispute category, and figuring out which fields disagree.

But the prompt didn't stop there. It also asked the model to compute:

- Net exposure: abs(MTM) − threshold - The correct call amount - The undisputed amount: max(0, internal_SIMM − collateral_on_hand)

Three dollar figures, computed inside the prompt, returned as LLM-generated numbers in the response schema. The exact anti-pattern our benchmarks exist to warn against, shipping in our own production API.

The fix

The redesign splits the endpoint along the line where language ends and math begins:

The LLM now emits only: the dispute category, the disputed field, an extracted CSA haircut rate (extraction, not computation), and an escalation target.

Deterministic Python computes: every dollar figure — net exposure, correct call amount, disputed amount, undisputed amount — from the firm's internal calculation and the notice.

The response schema is byte-identical. Callers see the same fields; the numbers just come from code instead of a token sampler. Every formula was validated against all 250 ground-truth cases before implementation, and the perfect-classifier results are now locked in as a regression test.

Measuring exactly what changed

The clean way to measure this isn't a fresh benchmark run — classification quality drifts between runs, and we wanted to isolate one variable. Instead, we replayed history.

We took 750 recorded runs (250 cases × 3 passes) of Claude Sonnet 4.6 on AAL-D-002, with the model's per-case classification outputs preserved. Then we scored the same classifications through both arithmetic paths. Same model, same cases, same classification decisions. The only thing that changes is who does the math.

correct_call_amount (benchmark-scored, ±$1,000): LLM arithmetic 80.4% → deterministic Python 84.7% disputed_call_amount (benchmark-scored, ±$1,000): 77.2% → 79.0% pay_undisputed: 92.8% → 95.9%

A 4.3-point gain on the primary scored field is real, but it's not the headline.

The headline is the ceiling

With a perfect classifier feeding the deterministic formulas, accuracy hits 95.6% on correct call amount and 92.8% on disputed amount — validated against ground truth and enforced as a regression test. Which means the remaining gap between 84.7% and 95.6% is entirely classification error: the model occasionally picks the wrong dispute category or field, and correct formulas applied to wrong inputs produce wrong outputs.

That decomposition is the point. Before the fix, an error could come from two entangled sources — a misclassification or a fabricated number — and you couldn't tell which. After the fix:

The error class changed. A wrong number used to mean the model invented a plausible-looking figure. Now, every dollar in the response is traceable: this classification input, through this formula, produced this output. Wrong-but-auditable beats wrong-and-hallucinated in any operations workflow where someone has to defend a number to a counterparty.

Accuracy work now has one lever. The arithmetic ceiling is fixed and tested. Every future point of improvement comes from better classification — a language problem, which is where model improvements actually land.

Fabrication risk went to zero. Not reduced. Zero. The model cannot output a dollar figure, because no dollar figure passes through it.

The uncomfortable part

The violation wasn't hidden. It was in a prompt we'd written ourselves, in an endpoint whose benchmark compatibility we advertise, adjacent to a rule we'd documented explicitly. It survived because the endpoint worked — 80% accuracy is good enough to demo, and nothing about a fabricated call amount looks different from a computed one.

That's the trap in every LLM workflow doing arithmetic today. The failure mode is silent, plausible, and only visible against ground truth. If we could ship it while writing benchmarks about it, so can anyone.

We'd suggest auditing your own pipelines for prompts that say "calculate," "compute," or "subtract." Ours did.

AAL-D-002 methodology and the full 250-case benchmark are documented in our research index. The discrepancy detection API — where the LLM extracts and Python computes, now everywhere — is live at aialphalabs.ai.