A failed test is a clue. Follow the evidence.
A practical way to move from a red CI job to a defensible root cause: separate observations, hypotheses, and proof.
Capture the failure before changing the system
A red result tells you that an expectation was violated. It does not yet tell you which component caused the violation. Preserve the first useful evidence: the assertion, timestamps, workload settings, software version, test revision, and the system state around the event.
For a storage timeout, distinguish the test runner’s deadline from the application’s error and the storage system’s own event record. A shared timestamp alone does not establish a causal relationship.
Write competing explanations
Keep at least three possibilities open: product behavior, test logic, and environment. Treat them as hypotheses that can be falsified, rather than labels to assign from the first log line.
| Hypothesis | Evidence that would help |
|---|---|
| Product regression | The same controlled workload differs between builds. |
| Test race | The assertion runs before the required state is ready. |
| Environment issue | An independent check also observes a host or path failure. |
Change one meaningful variable
Keep the workload and host configuration fixed while comparing two builds. Then hold the build fixed while changing the suspected environmental condition. Record both successful and failed runs; silently dropping retries hides the failure rate.
For intermittent failures, choose the repeat count and pass criteria before running the comparison. Report the counts and conditions, and avoid claiming that one clean rerun proves a fix.
Close the loop with a regression check
A useful root-cause report connects the trigger, the observed behavior, the faulty mechanism, and the verification. The regression should exercise that mechanism rather than merely repeat a broad scenario that happened to fail once.
This is an original starter checklist for the blog, not a report of an actual incident.