Green is a very persuasive colour.
A test suite passes. CI gives you a tick. The pull request looks safe. Everybody moves on.
We had enough green tests to become uncomfortable with how often the real system could still surprise us.
Some tests were checking the shape of the implementation rather than its behaviour. Some fixtures had copied production commands and then quietly drifted away from them. The test runner itself had been stabilised by forcing the whole suite through one fork with a 14 GB heap.
The signal said healthy.
The machinery producing the signal was asking for a second look.
A test can be correct about the wrong question
One of the easiest brittle tests to write is a source test.
Does this file contain this function call? Does this command string match this regular expression? Does this block of code mention the expected service name?
Those tests can be useful when the source text itself is the contract. Most of the time it is not.
If I refactor harmlessly, they fail. If production behaviour breaks without changing the text they care about, they pass.
That is an awkward combination: noisy when nothing is wrong, quiet when something is.
We had accumulated enough of these that test failures were sometimes telling us 'the implementation no longer looks how I remember' rather than 'the product no longer behaves correctly'.
The difference matters even more when agents are doing the implementation. A capable coding agent can satisfy a source-shaped assertion perfectly. That does not mean the user-visible contract exists.
Hand-copied fixtures were worse
A fixture that reproduces a production command by hand looks responsible. The test is self-contained. Everything is explicit.
It also creates a second copy of the contract.
When production changes and the fixture does not, the test can remain green while proving the behaviour of a command nobody actually runs.
We saw that pattern in both Agent Bridge and the Platform. Tests had confidence in a remembered version of reality.
The repair is not to make the copied fixture more accurate. It is to stop asking the copy to be authoritative. Where practical, tests should exercise the real builder, parser or runtime boundary and assert the result that matters.
That sounds obvious after the fact.
Most useful testing lessons do.
The 14 GB heap was another kind of green
The full suite had also become expensive to keep stable. One broad workaround forced Vitest into a single fork and gave Node a 14 GB heap.
The tests passed more reliably.
They also stopped telling us much about parallel safety and resource cleanup.
A workaround like that can be useful containment. It can get a team moving while the actual fault is still unknown. The danger is letting containment quietly become the definition of a healthy suite.
Once the suite needs extraordinary memory and global serialisation to stay green, the useful question changes. Which tests leave work behind? Which resources survive their owner? Which fixtures rely on global state or order? Which failures only disappear because nothing else is allowed to run beside them?
The workaround is no longer merely helping the tests pass. It is changing what the tests are capable of revealing.
Workarounds become part of what the test measures
This is the part that feels less like an engineering lesson and more like a human one.
We are very good at adapting to the metric in front of us.
If CI is red, make CI green. If a test flakes, retry it. If the process runs out of memory, give it more memory. If parallel runs interfere, run everything serially.
Each change can be a sensible temporary containment.
The danger is forgetting that the containment changes the question.
A serial suite cannot tell you whether the tests are safely parallel. A giant heap is less likely to expose retention. Retries turn nondeterminism into probability. A source assertion tells you about source shape.
The dashboard still says green. The meaning of green has moved.
We changed the review question
The most useful question is no longer 'does this test pass?'
It is 'what would have to break in the real system for this test to fail?'
If the answer is a formatting change, a renamed helper or a copied fixture, the test may be defending the wrong thing.
If the answer is that the actual user-visible or runtime contract no longer holds, the test is much more valuable.
For regression work, we also want the broken path first. Reproduce the failure against the bad behaviour. See the test go red for the reason we think it should. Then make the repair.
That keeps the disagreement alive before a capable agent gives us a convincing explanation of why its fix must work.
This is why green cannot be the goal
I still want green CI. I would be fairly unhappy without it.
I just do not want us to confuse a clean signal with a correct system.
Tests are instruments. Instruments need calibration. Their assumptions drift. Their fixtures age. Their workarounds can hide the very thing they were originally meant to reveal.
The same is true of business metrics, health dashboards and autonomy budgets. A number only earns trust by staying connected to the thing we care about.
Our test suite did not fail because testing is unreliable. It taught us something more useful: confidence needs maintenance too.
Green is a result.
The hard part is making sure it still means what we think it means.