Test Failover

A TestBubble recovers one protected application, or every application of a recovery plan, along a DR path that declares Test. It recovers onto the path's target site into renamed, isolated namespaces, then validates the application, writes a report, and removes everything it created. A test never changes production objects or replication.

Prerequisites

A test failover requires the following:

Path declares Test: The DR path lists Test in spec.actions and carries a spec.test block. See DR Paths.
Discovered applications: Only discovered applications can be tested. Managed (GitOps) applications report Unsupported.
Readiness: Each application runs at the path's from site and is not NotReady or Unknown. Tests cannot be overridden.
No concurrent run: No unfinished recovery action and no earlier unfinished test holds the application.
Isolated network for Multus VMs: KubeVirt VMs attached to Multus networks need an isolated NetworkAttachmentDefinition on the target, named on the path as test.isolatedNad: <namespace>/<name>. It must have no uplink to production networks. A DHCP server on it that serves the production reservations gives the VMs their production addresses (see Site Profiles).
Clones on the storage side: On simplyblock storage, the clones come from the Simplyblock Operator's TestFailover resource (storage.simplyblock.io/v1alpha2) on the hub: one per protected PVC, created by dr-hub. The operator resolves the newest replicated snapshot of the volume through the control plane, clones it on the target storage cluster, and places a bound PersistentVolume and PersistentVolumeClaim in the bubble namespace, with the source's volume mode and filesystem. Where the hub does not serve that resource, the storage publishes its replicated consistency points as VolumeSnapshotContents (label dr.simplyblock.io/replicated-snapshot=true), and dr-agent clones them.

A path with a test block looks as follows.

DR path with a test block

apiVersion: dr.simplyblock.io/v1alpha1
kind: DRPath
metadata:
  name: fra-a-to-fra-b
spec:
  from: fra-a
  to: fra-b
  planRef: fra
  actions: [Failover, Relocate, Test]
  test:
    mode: bubble
    isolatedNad: openshift-adp/drtest
    quotas:
      maxCloneCapacity: 2Ti
    recentWithin: 720h

The only test mode is bubble (the default). quotas.maxCloneCapacity limits the total capacity of the clones a test may create.

TestBubble Specification

Field	Description
`pathRef`	Name of the DR path to test along. Required.
`applicationRef.name`	The protected application to test. Exactly one of `applicationRef` and `planRef` is set.
`planRef.name`	A recovery plan whose applications are tested together, in priority order.
`cloneSource`	`latest-replicated-snapshot` (default) or `secondary-snapshot`.
`holdFor`	Keeps the restored application running for this duration after validation, for manual inspection.
`maxLifetime`	Upper bound for the whole test (default `24h`). When reached, the test is torn down.
`abort`	Set to `true` to stop the test and tear it down.

The specification is immutable after creation, except abort and holdFor.

Running a Test

A test of a single application:

TestBubble for one application

apiVersion: dr.simplyblock.io/v1alpha1
kind: TestBubble
metadata:
  generateName: orders-test-
  namespace: ramen-ops
spec:
  pathRef: site-a-to-site-b
  applicationRef:
    name: orders

A test of a recovery plan restores its applications in priority order into one shared bubble, so the applications reach each other as they do in production:

TestBubble for a recovery plan

apiVersion: dr.simplyblock.io/v1alpha1
kind: TestBubble
metadata:
  generateName: erp-stack-test-
  namespace: ramen-ops
spec:
  pathRef: fra-a-to-fra-b
  planRef:
    name: erp-stack
  holdFor: 1h

Create the object and follow it with the short name tbub:

Starting and watching a test

kubectl create -f testbubble.yaml
kubectl -n ramen-ops get tbub -w

Phases

Phase	What happens
`Pending`	Pre-flight checks run, and the state of production and replication is recorded.
`Cloning`	Bubble namespaces are created and one clone PVC is created for every protected PVC, from the latest replicated consistency point.
`Provisioning`	Network isolation is set up in the bubble.
`Restoring`	The application's captured Kubernetes objects are restored into the bubble, tier by tier.
`Validating`	The validation checks run.
`Holding`	Only with `holdFor`. Ends early on `abort`.
`TearingDown`	Everything the test created on the target is deleted, then production and replication are checked for changes.
`Completed`	The bubble is verifiably gone, whatever the outcome.
`Failed`	Teardown could not verify that everything was removed.

A test resumes where it stopped after a restart of dr-hub. The outcome of the test is in status.report.outcome, not in the phase.

Bubble Isolation

The bubble keeps the test copy away from production:

Namespaces: Each protected namespace <ns> is restored as <ns>-drtest-<id>. The bubble namespaces are listed in status.bubbleNamespaces.
NetworkPolicy: Every bubble namespace gets the drtest-isolation NetworkPolicy. It denies all traffic except traffic within the bubble, DNS, and ingress from dr-agent for health probes.
Network attachments: For each NAD a VM names, a NAD of the same name is created in the bubble namespace with the configuration of the path's test.isolatedNad. A reference to a NAD in another namespace is rewritten to that copy during the restore.
Services: LoadBalancer and NodePort Services are turned into ClusterIP Services and lose their external IPs.
Excluded objects: Routes, Ingresses, Gateways, HTTPRoutes, NetworkPolicies, and NADs from production are not restored.
Read-only capture: A test only reads the captured Kubernetes objects of production and can never overwrite them.
Health probes: Probe targets such as <svc>.<ns> are mapped to <svc>.<ns>-drtest-<id>. Targets the bubble cannot answer for (external names, IP addresses) are skipped as not exercised.

Validation Checks

Check	Meaning
`clones-bound`	Every clone PVC is bound within the path's clone quota.
`pods-ready`	All pods in the bubble are ready.
`vms-running`	All VirtualMachines in the bubble are running.
`guest-agent-connected`	The guest agent of each VM is connected. Warns only, because many guests run no agent.
`probes-healthy`	The application's `health.probes`, mapped into the bubble, pass.
`teardown-clean`	Nothing labeled with the test's ID remains on the target after the teardown.
`invariants-held`	Production and replication are unchanged between the start and the end of the test.

The results are listed in status.checks.

Holding and Aborting

With holdFor, the test stays in Holding after validation so that the restored application can be inspected. To end a test early, set abort:

Aborting a test

kubectl -n ramen-ops patch tbub orders-test-x7k2p --type merge -p '{"spec":{"abort":true}}'

Once anything has reached the target, any failure, an abort, or reaching maxLifetime moves the test to TearingDown.

Report and Outcomes

status.report holds the result:

Outcome	Meaning
`Passed`	The application came up in the bubble and all checks passed.
`Failed`	A step failed. The report names the first failure and the step.
`FailedInvariant`	Production or replication changed during the test. Overrides every other outcome.
`Unsupported`	The application cannot be tested safely in this release (see below).

The report also records the test point (the consistency point restored), the achieved RPO at test time, the estimated RTO (from all clones ready until the last restore workflow completed), the consistency (group or per-volume), the coverage (what was exercised and what was not), and warnings. When an archive is configured, the report is stored as JSON and PDF in S3, and the key is recorded in status.reportKey.

Unsupported Cases

The test reports Unsupported rather than risk a false pass or a connection to production for:

Managed applications: GitOps-delivered applications.
Multus without isolation: A VM on Multus networks while the path names no test.isolatedNad.
Host-bound devices: VMs with SR-IOV interfaces, host devices, or GPUs.

Clone Sources

latest-replicated-snapshot: The default. Clones come from the newest replicated consistency point on the target. For a consistency group, all members come from the newest point every member has. The replica's snapshot is never modified or deleted.
secondary-snapshot: Snapshots of the secondary volumes. Behind the secondarySnapshot feature gate and not functional yet.

Teardown

Teardown deletes every object the test created on the target site, including the bubble namespaces, then waits until nothing remains. It then checks that production and replication did not change. Finished tests are retained and pruned together with reports, as described in Monitoring.

Sync Applications

A test of a sync application runs the bubble on the stretch cluster, in the target zone. The clones come from the storage's newest replicated snapshots of that zone, the objects from a Velero backup of the application's namespaces that dr-agent takes at the start of the test, and the bubble's workloads are pinned to the target zone.

Coming soon

Hosted-cluster test mode: Tests inside a hosted cluster with mirrored test networks, MAC address regeneration, guest IP regeneration, and Route host rewriting.
Managed applications: Test restores of GitOps-delivered applications.