Skip to content

Operations

Every disaster recovery operation in simplyblock DR is a custom resource on the hub cluster. A RecoveryAction moves one application, or all applications of a recovery plan, along one declared DR path. Its status is a journal of every step, and a finished action produces a report. This page gives an overview of the actions and the rules they share.

Relocate and Failover

A recovery action has one of three kinds. For a move, the target is the to site of the action's DR path, and the path must list the kind in spec.actions. A Restart names no path, because it stays on the application's site.

Kind Use Final sync Data loss Rollback
Relocate Planned failover, failback, and permanent moves Yes None Yes, on timeout while the application starts on the target
Failover Unplanned failover when the source site is lost or unusable No Up to the achieved RPO No
Restart Restart in place after the application's storage cluster recovered from a suspension None None No

A failback is a Relocate along the reverse path. Every kind restarts the application. For a sync application, the sites are the zones of one stretch cluster, and dr-agent moves the application between them without Ramen. Moving running workloads without a restart is described in Relocate (Online).

RecoveryAction Lifecycle

A recovery action passes through the following phases in order. It resumes at the phase it stopped in after a restart of dr-hub.

Phase What happens
Pending The action was accepted.
PreFlight Checks that the path declares the kind, the application is on the path and bound, no other action holds it, it runs at the path's from site, and its readiness allows the action. A failure here changes nothing.
PreSource Runs the externalHooks.preSource hooks on the source site.
RamenHandoff The volumes are switched to the target site, after a final sync for a Relocate. For a Restart, the workloads are stopped, the volumes reconnected, and the tiers started in boot order on the same site.
TargetStarting Waits until the application is available on the target.
Workflow Waits for the application's health probes on the target (up to 15 minutes) and records the RTO.
PostTargetReady Runs the externalHooks.postTargetReady hooks on the target site, for example, DNS or load balancer cutover.
Confirming Completes the report.
Completed The application runs and is healthy on the target.
Failed The action stopped. The step that failed is in status.steps.
RolledBack A Relocate timed out while starting on the target and was returned to the source.

The RTO is measured from the creation of the action until all health probes pass. The action times out after spec.timeout (default 30m).

Readiness Gate and Override

dr-hub computes a readiness verdict for every application on every declared path (see Monitoring). Pre-flight uses it as a gate:

  • Ready or Degraded: The action runs. Degraded checks become warnings in the report.
  • NotReady or Unknown: The action is refused unless it carries an override.

An override is spec.override.reason, a text of 10 to 1024 characters that is recorded in the report. Creating an action with an override requires the override verb on recoveryactions, which only the dr-admin role grants. An action the path does not declare is never allowed, even with an override.

Per-Application Lock

Only one unfinished recovery action may hold an application at a time. A second action for the same application, or a plan action that includes it, fails pre-flight. Tests are also refused while an action is running.

Reports

Each finished action writes status.report: the operator who created it, the override reason, the RTO, the achieved RPO for a failover, hook results, probe results, warnings, the pre-flight readiness, and, for VMs on mapped guest networks, the expected and observed guest address of every interface (guests[]). With an archive configured, dr-hub stores the report as JSON and PDF in S3 and records the key in status.reportKey. See Monitoring.

Section Contents