One of our most complete features took the main bot offline for about 37 minutes.
The autonomy runtime had a separate production database path. The application knew how to open it. The configuration knew where it lived. The production database opener was deliberately strict: if the database file did not exist, it refused to create one behind the deployment system's back.
All of those choices were defensible.
The deployment tooling had simply never been taught that this new database existed.
After a release restarted the interactive service, the application looked for the configured database, did not find it and failed closed. Systemd restarted it. It failed again. And again.
The feature worked in the sense that its code did what we had designed.
The system did not work at all.
We had divided responsibility too neatly
The runtime was not supposed to create production databases. That rule exists for a good reason. Database creation and migration belong to the guarded rollout path so activation, ownership and schema changes happen in a controlled place.
The rollout path already knew about the other production databases.
When the autonomy database was introduced, we changed the runtime and configuration but missed the inventory owned by deployment.
The gap lived between two components, which is an inconvenient place for a checklist to notice.
Runtime tests can prove that a missing production database is rejected. Deployment tests can prove that known databases are provisioned correctly. Both can pass while the new database is unknown to one side.
That is exactly what happened.
Rollback did not save us
The first instinct during a bad release is usually reassuring: roll back.
We did.
It did not fix the outage.
The previous installed release already contained the conditional code that opened the autonomy database. The environment file was shared across releases and still pointed at the missing path.
So rolling the current symlink backwards changed the release and preserved the trigger.
That was a useful reminder that rollback is not time travel.
A deployed system includes configuration, data, permissions and external state. If the fault lives outside the release artefact, an older artefact can reproduce it perfectly.
The containment was deliberately boring
We created the missing database once using the application's sanctioned schema-creating path, under the correct runtime identity, then restored the intended current release and verified the interactive service stayed up.
That got the product back.
The real fix was not the manual creation. The rollout code had to learn about the configured autonomy database so fresh installs and guarded deployments would provision it before activation, just like the other managed databases.
We also tightened the configuration path handling afterwards so installation and rollout could not disagree about whitespace in the configured path.
The important change was ownership convergence: the thing responsible for creating production databases finally knew about every production database the runtime could require.
‘Merged’ is a poor definition of finished
This incident exposed a habit I recognise in myself.
When a feature is mostly code, I define completion around the code. The implementation exists. The tests pass. The pull request is reviewed. The release contains it.
Those are necessary milestones.
The person using the product does not experience any of them.
They experience whether the service starts after deployment. Whether configuration is valid. Whether a restart works on Tuesday. Whether rollback actually restores service. Whether a fresh machine can reach the same state without an operator knowing which hidden file to create.
That is a longer path than the feature branch.
Operational completeness is part of the design
It would be easy to turn this into another giant deployment checklist.
I do not think that is the main lesson.
The stronger question is ownership: when a feature introduces a new durable thing, which existing owner has to know about it for the whole lifecycle to work?
A database is not only a runtime dependency. It has to be installed, permissioned, migrated, validated, backed up or deliberately excluded, carried through rollout and considered during rollback.
A service unit is not only a file in the repository. Something has to install it, enable it, stop it and know whether it should exist on a particular host.
A configuration value is not only a parser input. It has to survive release changes and agree with the tools that consume it before the runtime starts.
Thinking in lifecycle ownership catches more than adding another box labelled 'deployment'.
The fail-closed behaviour was still right
There is a tempting post-incident fix that would have hidden the integration mistake: let the runtime create the database if it is missing.
That would have made this outage disappear.
It would also have weakened the production boundary that told us deployment was incomplete.
We kept the strict opener.
The failure was noisy and painful, but it was honest. Production state creation still belongs to rollout. The repair was to make rollout complete, not make the runtime quietly compensate for it.
That distinction matters to me because resilient systems can become very good at hiding ownership mistakes until the next, stranger failure.
Finished is the whole path
I used to think of deployment as the last stage after a feature was finished.
I think that language is backwards.
If a feature needs a database that deployment cannot create, the feature is not finished. If it survives a normal run but not a restart, it is not finished. If rollback preserves the fault and nobody has modelled why, the operational story is not finished.
The code can be correct inside its boundary and the product can still be unavailable.
For about 37 minutes, ours was.
That is enough evidence for a better definition of done.