Operations
Every disaster recovery operation in simplyblock DR is a custom resource on the hub cluster. A RecoveryAction moves
one application, or all applications of a recovery plan, along one declared DR path. Its status is a journal of every
step, and a finished action produces a report. This page gives an overview of the actions and the rules they share.
Relocate and Failover
A recovery action has one of three kinds. For a move, the target is the to site of the action's DR path, and the
path must list the kind in spec.actions. A Restart names no path, because it stays on the application's site.
| Kind | Use | Final sync | Data loss | Rollback |
|---|---|---|---|---|
Relocate |
Planned failover, failback, and permanent moves | Yes | None | Yes, on timeout while the application starts on the target |
Failover |
Unplanned failover when the source site is lost or unusable | No | Up to the achieved RPO | No |
Restart |
Restart in place after the application's storage cluster recovered from a suspension | None | None | No |
A failback is a Relocate along the reverse path. Every kind restarts the application. For a sync application, the
sites are the zones of one stretch cluster, and dr-agent moves the application between them without Ramen. Moving
running workloads without a restart is described in Relocate (Online).
RecoveryAction Lifecycle
A recovery action passes through the following phases in order. It resumes at the phase it stopped in after a restart of dr-hub.
| Phase | What happens |
|---|---|
Pending |
The action was accepted. |
PreFlight |
Checks that the path declares the kind, the application is on the path and bound, no other action holds it, it runs at the path's from site, and its readiness allows the action. A failure here changes nothing. |
PreSource |
Runs the externalHooks.preSource hooks on the source site. |
RamenHandoff |
The volumes are switched to the target site, after a final sync for a Relocate. For a Restart, the workloads are stopped, the volumes reconnected, and the tiers started in boot order on the same site. |
TargetStarting |
Waits until the application is available on the target. |
Workflow |
Waits for the application's health probes on the target (up to 15 minutes) and records the RTO. |
PostTargetReady |
Runs the externalHooks.postTargetReady hooks on the target site, for example, DNS or load balancer cutover. |
Confirming |
Completes the report. |
Completed |
The application runs and is healthy on the target. |
Failed |
The action stopped. The step that failed is in status.steps. |
RolledBack |
A Relocate timed out while starting on the target and was returned to the source. |
The RTO is measured from the creation of the action until all health probes pass. The action times out after
spec.timeout (default 30m).
Readiness Gate and Override
dr-hub computes a readiness verdict for every application on every declared path (see Monitoring). Pre-flight uses it as a gate:
- Ready or Degraded: The action runs. Degraded checks become warnings in the report.
- NotReady or Unknown: The action is refused unless it carries an override.
An override is spec.override.reason, a text of 10 to 1024 characters that is recorded in the report. Creating an
action with an override requires the override verb on recoveryactions, which only the dr-admin role grants. An
action the path does not declare is never allowed, even with an override.
Per-Application Lock
Only one unfinished recovery action may hold an application at a time. A second action for the same application, or a plan action that includes it, fails pre-flight. Tests are also refused while an action is running.
Reports
Each finished action writes status.report: the operator who created it, the override reason, the RTO, the
achieved RPO for a failover, hook results, probe results, warnings, the pre-flight readiness, and, for VMs on mapped
guest networks, the expected and observed guest address of every interface (guests[]). With an archive
configured, dr-hub stores the report as JSON and PDF in S3 and records the key in status.reportKey. See
Monitoring.