On 23 September we managed to turn a disk-space problem into an availability incident.
The production Agent Bridge service was healthy. A guarded rollout stopped it, moved into the disruptive part of the deployment, filled the filesystem, and failed.
Then the automatic rollback tried to recover.
It needed more disk.
There was none.
The result was much worse than "the new release did not deploy". The Telegram service remained offline until we recovered the previous runtime manually.
The embarrassing part was that rollback existed. It just was not part of the resource budget.
We crossed the dangerous boundary too early
The deployment already did a number of sensible things.
It staged an immutable release before activation. It verified release identity. It took database backups. It had a rollout sentinel, migration checks, acceptance checks and a restore path.
What it did not have was one answer to a simple question:
Before we stop the healthy service, can everything that remains — including recovery — fit?
Some disk checks existed deeper in individual components. They could tell us, for example, that a particular archive did not have enough headroom to install.
That is useful defence in depth. It is too late if the service has already been stopped.
The availability boundary was systemctl stop, not the eventual archive extraction.
The check needed to sit before that boundary and account for the whole remaining transaction.
Old safety evidence had become the hazard
The production filesystem also contained the history of our attempts to be careful.
Rollout evidence had grown to roughly 1.7 GiB. Database rollout backups had grown to roughly 1.4 GiB.
Neither category was junk. The logs existed so we could understand deployment failures. The backups existed so we could recover from them.
Unbounded, they had consumed the space needed to perform the next safe deployment.
There is a useful lesson in that which I keep encountering elsewhere: safety artefacts need lifecycle ownership too.
"Keep evidence" is not a retention policy. "Take a backup" is not a recovery policy.
We changed the rollout so proven terminal history can be pruned under a bounded policy while active, interrupted or ambiguous state is retained. The point is not aggressive cleanup. The point is to know which bytes still protect us and which bytes have become historical baggage.
Activation space was not enough
Our first pass at the fix still was not strong enough.
It is easy to calculate whether the new release itself will fit and call that admission.
A reviewer pointed out several ways we could still pass that test and fail after stopping the service: different destinations could live on different filesystems; a new database could allocate after containment; migrations could grow data; host components could need staging; and the previous release might need its own component work during rollback.
That changed the model again.
The question is not "is there enough free disk?"
It is "for every filesystem we are about to depend on, can the complete remaining transaction and the worst recovery path fit there with reserve?"
That is a much less convenient calculation. It is also the question the deployment is actually betting availability on.
Prepare while serving
The other improvement was to make the contained window smaller.
Anything expensive that can be prepared while the old version is still serving should be prepared then.
Download the release. Verify it. Extract large host components. Work out the required space. Prune safe historical state. Do as much as possible before the healthy process is touched.
Then the disruptive section becomes closer to what I had assumed it already was: final database consistency work, minimal activation, restart, acceptance, or restore.
This is one of those changes that feels obvious only once the incident has happened.
We had thought about deployment as a sequence of technically correct steps. The more useful model is a sequence of risk boundaries. The important question is what must still be true before crossing each one.
Rollback is part of the forward path
The biggest change in my thinking is about rollback itself.
I used to treat rollback as the emergency route. Build the happy path carefully, then make sure there is a way back if it fails.
But if the rollout is allowed to spend resources that rollback requires, rollback is not really a guarantee. It is a hope that enough of the environment survives the failed attempt.
Recovery capacity has to be reserved before the attempt starts.
That applies to disk, but probably not only to disk. Database compatibility, credentials, network access, previous binaries and operator entry routes can all be consumed or invalidated by the forward change.
A rollback plan that depends on those things should treat them as committed resources.
What I am watching next
We now make low-headroom admission a safe refusal before service containment, and we have kept tightening the calculation as production has exposed missing cases.
I do not expect a deployment system to prove every possible future failure away. That would become its own form of paralysis.
I do expect it to protect the obvious route home.
The compact version of the lesson is the title:
Rollback needs disk too.
More generally, a recovery path needs whatever it will depend on after the forward path has gone wrong.
Reserve that before you break the healthy thing.