Rollback Strategy
Triggers
Summary
A rollback is triggered by predefined criteria or responder judgement after automated or manual evidence identifies a deployment failure.
Standards
- An automated signal SHOULD be raised where a production threshold breach or critical security finding within a defined monitoring window is attributable to a deployment.
std-del-triggers-01 - A rollback SHOULD be triggered when a signal satisfies predefined rollback criteria or a responder determines that rollback is warranted.
std-del-triggers-02 - A rollback SHOULD NOT be triggered where the failing capability can instead be disabled through an existing feature flag's kill switch.
std-del-triggers-03 - A rollback MAY also be triggered based on a deployment's outcome identified through a means other than an automated signal, such as manual testing.
std-del-triggers-04
Related Standards
Implements These Principles
Rollback or Forward Fix
Summary
A rollback is the default response; a forward-fix is used where reverting would cause material loss of function or data or where the failure is unrelated to the latest deployment.
Standards
- A rollback SHOULD be chosen over a forward-fix by default, since reverting to a previously verified version is faster and carries lower risk than developing and verifying a new fix under time pressure.
std-del-rollback-or-forward-fix-01 - A forward-fix SHOULD be chosen over a rollback only where reverting would itself cause a material loss of function or data, or where the failure is unrelated to the most recent deployment.
std-del-rollback-or-forward-fix-02
Related Standards
Implements These Principles
Rollback Target & Timing
Summary
A rollback defaults to the most recently verified version, and completes within the service's recovery time objective.
Standards
- A rollback's target version SHOULD default to the version most recently verified through the service's own deployment history.
std-del-rollback-target-timing-01 - Selection of a rollback target older than the most recently verified version SHOULD be recorded with the rollback.
std-del-rollback-target-timing-02 - A rollback MUST be capable of being fully executed within the service's defined recovery time objective.
std-del-rollback-target-timing-03
Related Standards
Implements These Principles
Rollback Data Compatibility
Summary
A rollback proceeds only when its target version is compatible with the current data and schema state; otherwise, a forward-fix restores compatibility.
Standards
- A rollback MUST NOT proceed where doing so would run an incompatible version against the current data or schema state.
std-del-rollback-data-compatibility-01 - A forward-fix MUST be used until compatibility is restored where rollback would run an incompatible version against the current data or schema state.
std-del-rollback-data-compatibility-02
Related Standards
Implements These Principles
Rollback Scope
Summary
A coordinated rollback restores a compatible state across affected services, excludes unrelated services, and follows a defined sequence where order affects compatibility.
Standards
- A coordinated rollback MUST restore a compatible state across every service affected by the failing change.
std-del-rollback-scope-01 - A service unaffected by the failing change MUST NOT be included in the rollback.
std-del-rollback-scope-02 - Where rollback order affects compatibility, dependent services MUST revert in a defined sequence.
std-del-rollback-scope-03
Related Standards
Implements These Principles
Verification
Summary
A rollback is verified against the same automated health checks as any deployment before it counts as successful.
Standards
- A rollback MUST be verified against the same automated health checks required of any deployment, supplemented by manual testing where warranted, before it is deemed successful.
std-del-verification-01
Related Standards
Implements These Principles
Review
Summary
A rollback triggered by a production failure is reviewed according to its impact, and its recovery time is measured against the objective.
Standards
- A rollback triggered by a production failure SHOULD receive incident review where its impact meets the service's incident criteria.
std-del-review-01 - The time taken to complete a rollback SHOULD be measured against the service's defined recovery time objective, so a gap between actual and required recovery speed is identified and addressed.
std-del-review-02