When an AI Should Stop and Ask a Human — and Why "Please Clarify the Requirements" Isn't a Question
A look at the grounds on which an AI is obliged to interrupt an analysis and turn to a human, why "ask when you're unsure" doesn't work as a rule, what a question has to look like for it to be answerable, and how all our questions once quietly disappeared.
The analysis has been running for twenty minutes, and then it stops. On the screen is a question:
The specification names a grade of insulation that appears on none of the attached sheets. Continue, treating it as equal to the nearest analogue in the schedule, or wait for the missing sheet?
The first reaction is usually irritation: I paid for the machine to work on its own. The second, on reflection, is different. The alternative to this question isn't "the machine managed on its own". The alternative is the machine quietly picking one of the two and not saying which.
This article is about where the line runs between "the AI decides for itself" and "the AI is obliged to ask" — and why drawing it turned out to be harder than it sounds.
A question looks like a hold-up, but the thing to compare it against isn't a flawless analysis — it's a silent choice.
Why "ask when you're unsure" doesn't work
The most obvious rule — "let it ask whenever it's unsure" — doesn't work, and for a reason we've already been through: a model can't measure its own confidence. The number it produces in response to "how sure are you" is one more generated token, not a measurement.
An unpleasant consequence follows: a threshold of "ask below such-and-such confidence" can't be tuned, because there's nothing to tune. In practice you get two degenerate outcomes, and we've seen both.
Threshold too low — the AI asks about everything. Every ambiguity turns into a question, the analysis dissolves into two dozen clarifications, and the human spends more time on them than they would have spent checking the thing themselves. A tool that needs supervision at every step saves nothing.
Threshold too high — the AI never asks. And then it does what language models do by default: it fills the gap with something plausible. Not out of malice — a continuation of the text can always be found, whereas "there is no data here" isn't a continuation.
So the grounds for stopping have to be structural, not emotional. Not "I feel unsure", but "here is a specific condition under which going further isn't allowed".
Three grounds to stop — and one there shouldn't be
There turned out to be three conditions under which an analysis is obliged to break off.
First: the answer physically isn't in the documents provided. The specification refers to a sheet that isn't in the set. A calculation rests on a value given nowhere. The important thing here is not to confuse two things: "the data isn't there" is not the same as "I couldn't find it". The difference between them is the subject of the check, not grounds for a question; we wrote separately about telling one from the other. A question is appropriate when absence has been demonstrated, not assumed.
Second: two sources contradict each other and neither outranks the other. The plan says one thing, the section another, and nothing in the set makes either of them authoritative. The machine can surface the contradiction — but choosing between two documents of equal standing isn't its call: that stops being reading and becomes a design decision.
Third: the fork is commercial, not technical. "Should we work to the building code or to the client's brief where the two diverge" is not a question about the document, it's a question about what you agreed with the client. The machine has no access to that agreement, and it shouldn't invent one.
And here are grounds there shouldn't be: "just in case". A question asked for cover looks responsible, but it hands the work back to the human. It's the same temptation we described in the reviewing role: declaring a task unsolvable is safer than solving it. The difference is that this time the price of that safety is paid in somebody else's time.
Grounds for stopping have to be a checkable condition, not a feeling the model reports.
A question you can't answer isn't a question
The right grounds are half the job. The other half is how the question is phrased, and here language models behave badly by default.
A model told to ask questions readily produces polite formulas: "it is recommended that the finishing requirements be clarified", "please provide additional information". There's no answering that. It isn't clear what exactly is being asked, what the options are, or what changes depending on the answer.
A question that works is built differently, and it has three obligatory parts:
- What specifically is unknown — with an address: sheet, line, value. Not "there is an ambiguity in the specification" but "item 14 names a grade that doesn't appear in the schedule".
- What the options are — as a ready-made list wherever possible. Answering by picking one of two takes seconds; answering an unclear request in free text does not.
- What changes depending on the answer — "if the nearest analogue, quantities will be computed from it; if we wait for the sheet, this part of the analysis stays open". The human has to understand the price of their choice rather than guess at it.
Checking the phrasing is easy: read the question aloud and try to answer it in one sentence. If you can't, it isn't a question — it's a handover of responsibility dressed up in polite text.
"Please clarify the requirements" isn't a request for information — it's a request that you do the machine's work.
How our questions disappeared — and we didn't notice
Now the unpleasant part.
At some point the questions stopped. Not "became rarer" — they vanished entirely, across every kind of analysis at once. Nothing broke in the process: analyses ran, reports were produced, no errors in the logs.
The cause turned out to be a single line of instruction. The model had been told: before asking a question, record your current progress — and then a specific way of doing that was named. On some kinds of analysis no such method physically exists: they have a different set of instruments. The condition became unsatisfiable, and since the precondition couldn't be met, neither could the action. The model dutifully stopped asking questions everywhere.
The instructive part isn't the mistake itself but why it stayed invisible. An AI that has stopped asking looks exactly like an AI that has nothing to ask. An absence of questions doesn't look like a fault — it looks like smooth work. For a while we thought the analyses had simply got cleaner.
Hence a rule we took away, and it runs wider than our product: any function that manifests rarely needs a separate sign that it is alive. Not "no questions, so all is well", but a visible trace: the function ran and chose to stay silent is one state, the function never ran at all is quite another. Until you can tell them apart, the second one waits quietly for its moment.
Silence is never evidence. These are two different states, and you have to be able to tell them apart.
What we're not claiming
A pause costs money. A halted analysis waits for a human, and the human is busy. A question asked on Friday evening costs a weekend; that's why notifications about questions go out on every available channel regardless of how the other alerts are configured — if the user doesn't find out about the question, all the preceding work simply hangs. And for the same reason the number of clarifications is capped: an analysis that asks endlessly isn't an analysis.
The answer has to continue the work, not restart it. Our first version of the pause lost what had already been done, so after an answer the model re-read the set from scratch — and sometimes asked for the same files again. A question that resets progress costs more than the uncertainty it was asked about.
A question can go stale. If the analysis has taken a different route, an old clarification loses its meaning — and the worst case is that the human answers it anyway. An answer to an irrelevant question looks like perfectly good data and steers the analysis somewhere it shouldn't go.
A well-put question doesn't make the conclusion right. All it does is hand back a choice that would otherwise have been made for you in silence. What to do with that choice is still a human decision.
What to do about it
If you're evaluating a tool that "analyses documents on its own", ask two things.
What does it do when the data isn't sufficient? The answer "it always produces a result" isn't a virtue. It means the gaps get filled and you won't learn where.
What does one of its questions look like? Ask for a real example. If it's "please clarify the requirements", then questions as a feature aren't really there.
To see what an analysis looks like when every finding carries an address and a basis, and the uncertainties are set out separately, take a look at FIBRA's open demo report.
In the next article: arithmetic — why every calculation on our side is done by an ordinary engine rather than a neural network, and why that isn't over-caution.
For the technical audience: a couple of clarifications
- The pause happens at a step boundary, not in the middle of a chain of reasoning. That makes it far easier to guarantee that the state you return to after an answer is coherent: a chain of reasoning cut off mid-sentence doesn't resume meaningfully anyway.
- A question notification is the only kind of alert that ignores the user's channel settings. That's a deliberate exception: every other notification reports work that has finished, whereas this one reports work that will never finish without an answer. The only filter is whether a channel is verified — which is why, when you launch an analysis capable of asking questions, the system warns you about unverified ones.
- Clarifications don't consume the retry budget for the analysis itself. Otherwise a question would compete with the work for the same allowance, and the model would be penalised for having asked.