Arithmetic Isn't for Neural Networks — Why Every Calculation on Our Side Is Done by a Plain Engine

A language model gets three-digit multiplication wrong — and gets it wrong with confidence. A look at why that happens, why our model hands over a formula instead of an answer, the two rules that make the scheme work, and what it fundamentally cannot check.

The model found all three dimensions on the drawing correctly. It wrote them out: 4.54 × 3.89 × 12.18. Then it did the multiplication — and produced a number that reconciles with nothing.

This isn't a rare glitch or a sign of a poor model. It is normal behaviour, and understanding why changes how systems like this have to be built at all.

The model reads the dimensions correctly but multiplies them wrong — and nothing in the answer shows it

The nastiest property of a model's arithmetic error is that it looks exactly as confident as a correct answer.

A neural network doesn't calculate — it continues text

When you ask a model to multiply three numbers, it isn't performing multiplication. It is predicting which characters plausibly follow the equals sign.

For small numbers that works: "2 × 2 = 4" appeared in the training text so many times that there's nowhere to go wrong. For three-digit numbers with decimals it doesn't. There are far too many combinations, each specific one occurred rarely or never, and the model assembles an answer by pattern: the right number of digits, a plausible magnitude, a similar-looking ending.

That's exactly why models' arithmetic errors look so odd. It isn't random scatter around the correct value — it's a plausible forgery. The order of magnitude is usually right, often the first digit too, the length of the number fits. The error sits in the middle, where the eye doesn't look for it. Obvious nonsense would be safer: you'd spot it at once.

It's important not to conflate two different problems here. An invented clause number of a building code can be caught by checking against a document database: either the clause exists or it doesn't. A multiplication error has nothing to check against — except by computing it again. And asking the model to recompute is pointless for the same reason asking "are you sure" is pointless: the second time it takes the same route.

In a documentation audit this matters. The whole point of checking quantities is to independently recompute what's printed in the table and compare. If the recomputation is done by something that gets multiplication wrong, the check becomes a lottery: a discrepancy may mean an error in the document, or an error by the checker.

The fix: the model hands over a formula, not an answer

The way out turned out to be simple, though not immediately obvious. You have to separate two jobs that intuitively look like one.

The model's job is to find the operands. Open the sheet, work out which dimensions the value is built from, locate them on the drawing, account for notes and factors. This is precisely what a language model is good at: reading, matching, understanding context.

The machine's job is to compute. The model doesn't write the result. It writes an expression — 4.54 * 3.89 * 12.18 — and that expression is evaluated by an ordinary arithmetic engine. The same kind that's in a calculator: deterministic, no variation, always identical on identical inputs.

The server then compares the computed value against what stands in the document's column and reports the difference. A discrepancy within rounding isn't a finding. A discrepancy by a factor is a finding with an address.

One detail matters more than it seems: the engine doesn't execute code. It parses an arithmetic grammar — numbers, four operations, parentheses — and nothing else. The temptation of "let the model write a script and we'll run it" exists, but that's a fundamentally different level of risk: running what a model generated on your own server isn't worth it for a task that comes down to addition and multiplication.

Division of labour: the model finds the operands and composes the expression, a deterministic engine evaluates it

The model does what it's strong at — reading. The computing is done by something that doesn't make mistakes.

Two rules without which the scheme doesn't work

The idea is simple, but in its first version it had almost no effect. Two rules were needed, and neither is obvious in advance.

Rule one: don't compute what was simply read. The model happily wrapped a formula around a value it had taken from the table directly: 762.9 → 762.9, discrepancy zero. Formally everything is right, the engine ran, the diagnostics gained one more calculation. Nothing whatsoever was verified: the number was checked against itself. Empty calculations like these clutter the output and create a false sense of coverage — "fifty items computed", forty of which were computed against themselves. A formula belongs where the value is reconstructed from other data, not where it's printed.

This comes not from laziness but the opposite: the model conscientiously carries out "compute the quantities", and on a row where there is nothing to compute it produces the only available action — it copies the number across. The rule had to be phrased not as "compute more" but as "don't compute where there's nothing to verify".

Rule two: the calculation belongs in the calculation field, not in the text. This looks like a formality, but it's exactly what cost us our most spectacular finding. The model wrote in the finding's description: "4 × 3.7 × 12.95 = 191.66 m²" — and noted that the specification says 2,192.50. The calculation was performed, the conclusion correct, the elevenfold overstatement detected. But the expression stayed as prose inside the text rather than as a value in the formula field — which means the engine never recomputed it, and nobody confirmed that 4 × 3.7 × 12.95 really does give 191.66. The finding was right in substance and unverified in form.

The remedy is narrow: if the text of a finding already contains a multiplication with a result, then the model could and did compute — it simply put the result in the wrong place. The server notices this and asks for the expression to be moved into the formula field. Crucially, the requirement fires only when a calculation has already been written out: pressing "compute" on findings that aren't computable is pointless and harmful.

Two rules: a formula only for reconstructed values, and the calculation lives in the calculation field rather than the description

A calculation left as prose looks like verification — but nobody verified it.

What the engine doesn't check

Now the honest part, without which all of the above would sound better than it is.

The engine guarantees exactly one thing: that the arithmetic was performed correctly. It does not check the operands.

If the model took a dimension of 3.7 for 3.1 — misread a digit on the drawing, confused a callout, picked the dimension off an adjacent detail — the engine will faithfully multiply by 3.1 and return an impeccably computed wrong answer. The arithmetic is flawless, the number is wrong, and nothing about the calculation itself shows it.

This is a fundamental limit, not an unfinished feature. There is only one way to verify an operand: look at the sheet again — that is, do the very reading work whose reliability the whole thing was built for. So operands have a different line of defence: they must be written out explicitly, with a note of where they came from, so a human can check them against the drawing in a minute without reassembling the calculation. The second role, deliberately built to argue, works here too: its question "and where did 3.7 come from?" is exactly about operands.

The engine verifies the arithmetic but not the source of the numbers: a wrong operand yields an impeccably computed wrong answer

A deterministic calculation removes one class of error entirely — and leaves the neighbouring one untouched.

What we're not claiming

This doesn't make the model "able to do arithmetic". It still gets multiplication wrong; we simply stopped asking it to. If language models learn to compute reliably tomorrow, the scheme won't change — because the point isn't only accuracy but verifiability: an expression can be read and re-checked, whereas a number obtained by unknown means cannot.

We don't show the formula in the report. The report carries the result and the discrepancy: the client needs a conclusion about the document, not our internal kitchen. The operands stay in the analysis and must be named — a finding that doesn't say where its numbers came from doesn't count as proven.

Not every value can be recomputed. Sometimes the documentation simply doesn't contain the data a quantity derives from. That's a separate kind of defect — the client can no more verify the quantity than the AI could — and it's more honestly recorded as a finding that "the value rests on nothing" than as an invented calculation.

What to do about it

The practical conclusion is short: if a tool shows you a computed number but doesn't show what it was computed from, treat that number as unverified. Whether it's right or wrong doesn't matter: you have no way of telling, and the model looks equally confident in both cases.

The working test is whether you can reconcile the operands against the sheet in a minute. If you can't, what you got isn't a check — it's one more opinion.

The same question is worth asking of your own processes, where a human works the calculator. The difference is that a human who miscalculates usually feels a flicker of doubt and re-checks, whereas a model feels nothing and therefore never re-checks. "Show me what this was computed from" is useful in both cases, but in the second it's the only thing that works at all.

To see what an analysis looks like when every value has its source named, take a look at FIBRA's open demo report.

In the next article: how an AI auditor games the system — not out of malice, but because any metric you set it is easier to optimise than the task itself.

For the technical audience: a couple of clarifications

  • The engine accepts an arithmetic grammar: numbers, four operations, parentheses. No variables, no functions, no reaching for external data — not out of asceticism, but because every capability you add turns "evaluating an expression" into "executing someone else's code", with the consequences that implies.
  • The calculation runs on every path that assembles a result, including the one where the model finishes directly without calling any instrument. That isn't belt-and-braces: on precisely that path calculations quietly went unevaluated for a while — the last step bypassed the single place where the engine was invoked.
  • Comparison against the printed value carries a rounding tolerance. Without it every other row of the table would become a "discrepancy" over the third decimal place, and the useful signal would drown in noise.