Recovery is a designed product behavior. It is not a collection of emergency commands added after deployment.
Recover the same operation first
Every effectful apply journals intent before dispatch and records attempts, fences, acknowledgements, observations, and reconciliation. After interruption, One resumes the same durable operation and determines the next safe action.
If dispatch may have crossed the boundary, the outcome is unknown. Recovery queries the provider, imports authenticated observations, and reconciles under the original fence. It does not turn ambiguity into failure so a duplicate can be sent.
Rollback moves forward
Rollback is a new plan toward a prior desired semantic state. It can require a new artifact, data conversion, provider transition, routing change, grant, approval, compensation, and postcondition evidence.
Some effects cannot be reversed. Their recovery contract may compensate, notify, quarantine, repair, or require human resolution. History retains both the original effect and the response.
Restore requires lineage
A recoverable snapshot identifies source logs or data ranges, schema and provider revisions, encryption, integrity root, retention, and compatibility. A restore plan proves that the target can interpret the snapshot and that later facts are replayable, intentionally discarded, or independently recovered.
Backup completion and restore readiness are different claims. Production policy can require periodic destructive restore exercises in an isolated environment and retain their exact evidence.
Failover is a state transition
Stateless rerouting can use health and load evidence inside a pre-authorized provider envelope. Stateful failover additionally requires replication position, fencing, promotion authority, continuity semantics, routing, split-brain prevention, RPO, RTO, and repair.
Health is evidence, not authority. A healthy unselected provider cannot receive traffic, and an unhealthy observation alone cannot grant another provider access to state.
Control planes may fail independently
Data planes retain the minimum authority and configuration needed to honor their availability contract when a planner, controller, registry, or network partition is unavailable. Offline operation is bounded by signed leases, cached policy, local resource limits, and an explicit reconciliation path.
High-value or irreversible effects default to online authorization. A plan cannot claim disconnected continuity unless every necessary provider, credential, data path, time source, and recovery action has executable evidence.
Disaster recovery is a complete root
A disaster-recovery strategy closes code, artifacts, data, configuration, secrets, identities, providers, quotas, routes, authority, observations, and operator access in the recovery environment. Regular drills execute that closure and compare observed RPO and RTO with policy.
The result is inspectable: what survived, what was rebuilt, which evidence was used, what data was lost by contract, which actions required human judgment, and whether the recovered system satisfies its semantic postconditions.
Example
Illustrative end-product commands resume one recovery operation:
one plan recovery release:sha256:… --checkpoint checkpoint:sha256:…
one apply plan:revision:sha256:…
one reconcile recovery:operation:sha256:…Follow Restore, fail over, and roll back.
Canonical owners: Deployment and History.