Restart After a Storage Recovery
A simplyblock storage cluster suspends when too many nodes or devices fail at once, and it comes back by itself once
enough of them return. The applications on the site did not move, but they did not recover either: their workloads
failed during the suspension, in no particular order, and their volumes need a clean reconnect. A RecoveryAction of
kind Restart brings such an application back on the same site, in boot order, with its hooks and health probes, and
with a report. With autoRestart enabled on the plan, dr-hub creates the action itself.
Automatic Restarts
dr-agent watches the StorageCluster objects of its cluster and records every return from Suspended to Online
or Degraded. A recovery counts once the cluster has stayed up for stableFor (default 2m). dr-hub maps every
protected application to its storage cluster through the cluster_id parameter of its StorageClasses and creates one
Restart per application and recovery when all of the following hold:
- Opt-in: The plan has
spec.autoRestart.enabled: true, and the application is not annotateddr.simplyblock.io/auto-restart: "false". - Scope: The application runs on the site of the recovered storage cluster and has a volume on it.
- Recent: The recovery is at most one hour old.
- Not concurrent: No other action for the application is unfinished, and no Restart was created for this recovery before.
spec:
autoRestart:
enabled: true
stableFor: 2m
The created action is named restart-<application>-<hash>, annotated with the storage cluster, the recovery count,
and the recovery time, and stamped created-by: dr-hub (storage recovery).
Starting a Restart by Hand
A Restart names an application and no path, because source and target are the application's current site. Readiness does not gate it, because the application is down. Managed applications are refused.
apiVersion: dr.simplyblock.io/v1alpha1
kind: RecoveryAction
metadata:
generateName: orders-restart-
namespace: ramen-ops
spec:
kind: Restart
applicationRef:
name: orders
Phases
- PreFlight: The application runs on the site that recovered, the storage cluster is
OnlineorDegraded, and no other action runs. - PreSource: The
preSourcehooks run, best effort, as on a Failover. - Restart: A dr-agent task on the site stops the workloads all at once (Deployments and StatefulSets scaled to zero, VirtualMachines halted), waits until no pod uses the application's volumes (10 minutes at most), reconnects the volumes, and starts the workloads tier by tier: each tier is scaled back, and its ready conditions must pass before the next tier starts. Workloads no tier selects start last.
- Workflow: The application's health probes.
- PostTargetReady: The
postTargetReadyhooks announce the application again. - Confirming: The report is completed. The action ends in
Completed.
A Restart that fails leaves the application stopped and reports the step it failed in. There is no retry loop: the next recovery, or an operator, starts the next one.
Report
status.report.restart holds the storage cluster, the recovery count, the time of the recovery, the seconds from the
recovery to the health probes passing (recoveryToReadySeconds), whether the volumes were reconnected
(volumesReconnected), and every step with its result and duration.
Coming soon
The volume reconnect (a demote and promote of every volume through the Simplyblock Operator) waits on the operator. Until it is available, a Restart stops and starts the workloads in boot order only, and its report says that the volumes were not reconnected.