In late September we had one of those production failures that makes a simple architecture suddenly feel non-negotiable.
The Farstax Platform was running a current, healthy revision.
A maintenance command launched from that environment loaded an older copy of part of the Platform lifecycle from /opt/agent-bridge.
Both copies existed on the same machine. Module resolution picked the stale one.
The deployed application was current. The lifecycle code acting on it was not.
That was the point where our deployment design stopped looking like a collection of useful mechanisms and started looking like competing authorities.
A healthy revision was not enough
We had immutable application revisions, health checks and rollback state.
We also had appliance code under /opt, resident maintenance commands, break-glass entry routes and scripts capable of converging the machine.
Individually, each piece had a reason to exist.
The problem was that more than one place could become the source of truth for the operation that changed the system.
The stale-source bug made this literal. The active Platform checkout contained the fixed maintenance template. The installed unit remained stale because the runtime resolved a lifecycle module from the old /opt tree.
It is difficult to reason about rollback when "which code is performing the rollback?" can have two answers.
We fixed that immediate source-authority defect, but the more useful response was to simplify the deployment model around it.
One implementation, several ways to reach it
The target we settled on was deliberately boring.
A qualified release contains an immutable Platform workload artifact. The resident lifecycle verifies and activates that artifact. The ordinary route and the break-glass route both end up invoking the same deployment implementation.
Break-glass is allowed to solve a different problem: secure entry.
It can establish the tightly bounded SSH route, verify the host identity and constrain the command that may be run. Once it reaches the machine, it does not get its own clever version of deployment.
That separation matters.
If the normal deploy path is broken because the workload is unhealthy, a recovery route that merely calls back into the unhealthy workload is not much of a recovery route.
If the break-glass path contains a separate build, activation and rollback implementation, we now have two deployment systems that can drift apart.
The useful middle is a resident lifecycle whose job is specifically to deploy the workload, with multiple authorised entry routes converging on it.
Build once became more important than build location
Before this change, production deployment could reconstruct a candidate using Git and npm ci.
That was safe enough in the narrow sense that we verified what we were activating. It was still asking production to recreate the release from source.
That made network availability, package resolution, disk state and source reconstruction part of the deployment transaction.
We moved those responsibilities into release publication instead.
The qualified release now carries the runnable workload bytes and their identity. Production verifies and materialises those bytes. It does not need to rediscover how to build them.
The phrase we kept returning to was:
Build once, qualify once, deploy immutable bytes, verify once.
It is not novel deployment theory. It was simply the right correction to a system that had accumulated too many ways to recreate itself.
We also separated unrelated release streams
Another source of fragility was that deploying the Platform workload could trigger broad appliance installation after the new workload was already healthy.
That meant an unrelated substrate problem could turn a successful application deployment into a reported deployment failure.
So we removed that coupling.
Platform workload activation owns Platform workload activation. Platform appliance convergence owns appliance state. Agent Bridge release owners own Agent Bridge rollout.
A release can contain identities for more than one component without pretending that every component must mutate every time.
Again, the simplification is mostly about being able to answer "who owns this?" without tracing half the machine.
Recovery is a design property, not an emergency script
I used to think of break-glass mainly as an operational feature.
Have a secure way in. Keep a rollback command. Document the procedure.
The work in September changed that for me.
Recovery quality is determined long before anyone types the emergency command. It depends on whether the recovery code has an independent enough footing, whether it consumes the same immutable release contract as normal deployment, whether the previous state is still available and whether all routes agree on what "active" means.
A clever emergency script cannot repair an ambiguous ownership model.
It will simply choose one of the ambiguities under more pressure.
What I am watching next
There are still edges here.
The appliance itself has a lifecycle. Release provenance has to be checked somewhere. Old revisions need pruning. Recovery state has to survive interruption. The protected Farstax workspace and customer workspaces do not even have identical topology.
I do not expect one generic deployment abstraction to make all of that disappear.
The rule I want to preserve is narrower:
The workload being replaced should not be the only thing capable of replacing or recovering it, and every authorised route should converge on one deployment owner.
That gives normal operation and break-glass different jobs without giving them different truths.
It took stale code being loaded from the wrong directory to make that boundary obvious.
I would rather not need that lesson twice.