Test the recovery, not just the interruption.
Fault injection gets interesting after the fault. Define what recovery means from the host, data, and system perspectives.
Define the contract first
Before injecting a fault, write the system’s expected behavior for that specific configuration. Is temporary I/O delay acceptable? What maximum recovery time is agreed? Which errors should the application observe? Those answers come from the product’s contract, not a generic idea of high availability.
A green health indicator is useful, but it cannot replace checking the application’s outcome and the integrity of its data.
Observe the full timeline
- Baseline: confirm the workload, paths, and health before injection.
- Fault: record what actually changed and when.
- Transition: measure application errors, latency, and relevant state changes.
- Recovery: verify restored behavior against the agreed contract.
- Afterward: validate data and check for lingering degraded conditions.
Use clocks with a known relationship when correlating events across machines. Keep uncertainty visible if timestamps cannot be reliably aligned.
Use independent checks
Combine workload observations with system evidence. A test that declares recovery from the same status field used to trigger the next step can miss a misleading status transition.
Consider validating checksums or application-level invariants where they fit the workload. Record the exact validation method: a successful read is a different claim from a comparison against expected data.