Not every failure is obvious. In large-scale systems, some issues do not throw errors, trigger alerts, or create visible customer impact right away. They pass quietly behind green dashboards and successful retries.
I think of these as "dogs not barking." The phrase comes from Sherlock Holmes, where the important clue was not what happened but what did not. A guard dog stayed silent when it should have reacted.
Here are two real examples I encountered.
The server that was not supposed to take traffic
It is common to keep a small subset of production servers fully provisioned but weighted out of front-end traffic. They provide a production-like place for validation without serving customers through the normal entry point.
One day, we discovered that these servers were receiving real requests anyway. The traffic did not arrive through the load balancer. Internal services discovered endpoints through a gossip mechanism and called them directly.
No alarm fired. The servers responded normally. The system did not break. The assumption that these hosts were isolated had simply stopped being true.
We found it while investigating something else. The immediate fix tightened service discovery and enforced stricter boundaries. The deeper lesson was that traffic does not always enter where the architecture diagram says it should.
On AWS, flow logs, service-mesh telemetry, target-group membership, and explicit security-group paths can help verify the real network. The EKS microservices architecture is a useful example because ingress, services, pods, and node networking create several possible paths.
The retry that hid the real bug
Our pre-production tests allowed automatic retries. That choice reflected reality because distributed systems are noisy and many clients retry failed requests.
Then one test failed on its first attempt and passed on retry. The pipeline marked it green and continued. In production, customers began receiving immediate 4xx responses for valid requests. Some clients retried and recovered. Others did not.
The bug was an uninitialized header object on first use. The retry ran on a warmed thread, where initialization had already happened, so it passed.
The retry did not create the bug. It removed the evidence.
We did not respond by stopping every deployment after any transient test failure. At scale, that would create a different reliability problem. We did begin treating "passed after retry" as its own signal that could reveal cold-start, initialization, ordering, or hidden state defects.
Broken assumptions need monitors too
In both examples, the software worked just enough to appear healthy. The important engineering question is: how would we know if this assumption stopped being true?
That can lead to useful checks:
- Assert that weighted-out hosts receive no production traffic.
- Track first-attempt failures separately from final retry success.
- Test cold paths deliberately.
- Reconcile expected and observed service membership.
- Alert on impossible states, not only elevated errors.
- Centralize logs so one green component cannot hide the end-to-end story.
The centralized AWS security logging architecture shows the value of preserving evidence away from the system that generated it.
Sometimes success that arrives a little too conveniently is worth a second look.
