The feature everyone wants from an autonomous agent is simple to describe: keep going.
Do not wait for me after every step. If the next action is obvious and allowed, take it. If new evidence arrives, adapt. Continue until the outcome is reached or you genuinely need me.
That is also the feature that can turn a helpful agent into an expensive loop.
We have been learning that autonomy is not mainly about teaching the system how to continue. Capable models are already quite good at finding another thing to do. The harder part is deciding when continuation is no longer allowed.
We found this in a tiny race
One of our autonomous-run bugs was almost comically specific. It was caught during review, before we had evidence of it happening in production, then reproduced with a red regression test.
We had a fixed cycle budget. Imagine the agent is allowed to run three times.
The third and final run is still in progress when an unhealthy observation arrives. The observation creates a reason to wake the agent again. The final run completes and the cycle budget is now exhausted, but the pending wake is already sitting there.
Without another budget check at the point where work is claimed, the system could take that wake and start a fourth run.
Four runs from a budget of three.
Nothing about the agent's reasoning caused the problem. The error lived in the seam between 'new work exists' and 'you are still allowed to do work'.
That is the sort of review finding I find more useful than a hundred architecture diagrams. It tells you what the architecture actually is.
A budget is not a number
Before that finding, it was easy to think of the cycle limit as a counter. Three means three.
Afterwards it became clear that a budget is a rule that has to hold at every path where work can begin.
A wake can arrive while work is in flight. A process can restart. Two observations can describe the same underlying event. A previous attempt can have recorded its intent but not its outcome.
If any one of those paths can create a new run after the budget is spent, the budget is decorative.
This is why autonomous systems become state problems very quickly. Not because the models need elaborate state machines, but because 'keep going, only while these conditions remain true' has to survive interruptions, retries and bad timing.
The edges are part of the product
There is a tendency to talk about limits as safety plumbing added around the interesting part. I am starting to think the limits are the interesting part.
A useful autonomous episode needs several exits. It can succeed because authoritative evidence says the outcome is reached. It can stop because its budget is exhausted. It can stop because the useful next action is outside its authority. There may be others: cancellation, contradictory instruction, a missing dependency or a system failure.
The important thing is that 'I can think of another action' is not itself a reason to continue forever.
That sounds obvious. Models are exceptionally good at thinking of another action.
Human supervision changes shape
This is also changing what 'human in the loop' means for me.
I do not want to sit inside every loop. If the agent needs approval after each step, I have built remote-controlled automation, not autonomy. But I do want to define the edges of an episode and know the system will respect them.
That leads to a different pattern. I approve a meaningful outcome and a bounded episode. The agent does the routine reasoning and implementation inside it. At the end, I get a concise account of what happened and, at most, one proposed next episode.
If I approve that, a fresh episode starts with fresh evidence.
The human is between episodes, not between tool calls. At least, that is the pattern we are testing now.
Why I prefer this to ‘run until done’
'Run until done' is wonderfully simple if 'done' is perfectly measurable and every intermediate action is harmless.
Real company work is rarely that clean. A goal can be long-lived. Evidence can change while the agent is working. An apparently sensible next step can require a new account, new spending or a new permission. A model can get stuck in a locally reasonable loop.
Putting an episode boundary around the work gives us somewhere to re-read reality. Did the outcome move? Did the previous action work? Has the situation changed? Is the next step still inside the authority we granted?
That pause is not necessarily a weakness. It can be a control surface.
We are not finished with this
I do not know the right episode size. Too small and I become the scheduler; too large and the budget stops being a meaningful constraint.
I also do not know whether owner approval between episodes is a permanent product shape or training wheels for the experiment.
I do know that a review found a path to one extra run behind a perfectly reasonable health observation, and the red test proved it. That was enough to change my mental model.
The stopping rule is not scaffolding around the autonomous system. It is part of the autonomous system.
If the next experiments show our current edges are in the wrong place, fine. At least we now know where to look.