Coding agents are excellent at saying 'done'. This is not a complaint. Humans are excellent at it too.
The agent reads the code, makes a change, runs a test and gives you a neat account of what happened. The explanation can be detailed enough that it feels like evidence.
Sometimes it is. Sometimes the software is still broken.
A lot of our recent work has been about learning not to confuse a good account of reality with reality.
The first version of truth is usually prose
When an agent finishes a task, the easiest thing for the surrounding system to consume is the agent's answer. It says the bug is fixed, the application recovered, the message was delivered once, the new flow works.
If you are building an autonomous loop, those sentences are dangerously convenient. You can feed them straight back into the system and let the next step begin.
We have become increasingly reluctant to do that.
The model's answer is useful evidence about what it tried and what it believes. It is not always the evidence that should close the loop.
A duplicate message made this very obvious
We spent time chasing a Telegram problem where a response could appear more than once while an agent was streaming and then finishing its answer.
The internal records looked sensible: one user turn and one assistant turn. That was not enough because the thing we cared about was what the person actually saw in Telegram.
A database could prove that the conversation had one logical answer. It could not prove that the interface displayed one message rather than two.
Eventually the useful evidence was embarrassingly direct: run it live and watch the message update in place.
Different question. Different source of truth.
That incident has stayed with me because it punctured a very engineering-shaped habit: using the evidence that is easiest to collect rather than the evidence that answers the question.
Health checks taught us the same lesson
Suppose an agent investigates an unhealthy application, restarts something and reports that the service is back. Should the company mark the incident recovered?
Maybe not.
A successful restart proves the restart command succeeded. It does not prove the application is healthy. The agent saying 'healthy now' proves the agent believes it is healthy; that may still not be the right measure.
If the original problem was detected by an independent health observation, the cleanest close is often a later independent health observation.
That creates a useful separation: the agent owns the attempt; the world owns the result.
This sounds obvious until you automate continuation
When a human is supervising closely, weak evidence is survivable. You read the answer, glance at the application, notice something odd and ask another question.
Autonomous continuation changes the cost. If 'I fixed it' becomes the condition for starting the next cycle, the agent's confidence can mutate the state of the system. A mistaken conclusion is no longer just a bad sentence. It becomes a control signal.
So we have started treating outcome evidence as its own design problem.
What observation would convince us if the agent were not in the room? A test result? A merged pull request? A health check? A user completing the workflow? A message visibly appearing once?
The answer depends on the goal. There is no universal 'done' event.
The company experiment makes this harder
Our current company goal is deliberately tied to a real-world outcome rather than agent activity.
An agent can improve the landing page, fix onboarding, open issues, write code or propose a campaign. None of those actions means the company goal has been achieved. The outcome moves only when the external evidence moves.
This is inconvenient. It means an impressive run can end with zero measurable progress.
Good.
If the goal is to learn whether autonomous work creates useful business outcomes, we should not let the agent grade its own homework.
What I now want from an agent report
I still want the narrative: what did you see, what did you think it meant, what did you decide, what did you do, what happened next?
That is valuable context for the next run and for me. But I want the report to sit beside the evidence, not replace it.
The distinction also makes failure less dramatic. An action can be reasonable and still not work. That is not a contradiction; it is information. The next agent should see the failed attempt and adapt rather than rewrite history so the previous action looks successful.
The phrase we keep coming back to
Internally, the rule is becoming something like this:
Agent prose can explain the outcome. It should not manufacture the outcome.
I expect exceptions. Some goals are inherently qualitative and will need judgement. Some evidence will be expensive or ambiguous. Sometimes the model may genuinely be the best observer available.
We have not solved that.
But whenever we can anchor 'done' to something outside the agent's own confidence, the system gets easier to trust. For this project, that is worth more than a beautifully written completion message.