Volume Migration

The Simplyblock Operator can move a volume's backing logical volume from one storage node to another while the volume stays online (live migration). A migration relocates a logical volume (and its snapshots). It does not move the storage node itself. This is different from Migrating a Storage Node, which relocates an entire storage node identity to a new host.

A volume migration moves only the logical volume itself, not the actual data. Since data remains distributed in the back storage, volume migrations is an online live migration and nearly instant. If node affinity is turned on for a cluster, back storage data realignment happens via rebalancing as an asynchronous task after one or more the volume migration(s) finished.

Volume migration is used in three ways:

Manual migration: requests a specific volume to move to a specific target node.
Drain / removal migration: the operator automatically evacuates a node's volumes before it is removed.
Auto-rebalancing: the operator continuously moves volumes off overloaded nodes.

All three paths share the same backend migration mechanism and the same post-migration data realignment.

Enabling Volume Migration

Volume migration is controlled per cluster by StorageCluster.spec.volumeMigrationSettings.

Volume migration settings

spec:
  volumeMigrationSettings:
    enabled: true                       # default: true
    dataRealignment:
      enabled: true                     # default: true
      interval: 10m                     # default: 10m

Field	Default	Description
`enabled`	`true`	When `false`, the operator does not act on `VolumeMigration` resources for this cluster.
`dataRealignment.enabled`	`true`	Enables automatic post-migration data realignment.
`dataRealignment.interval`	`10m`	How often the operator checks whether a realignment is pending.

Manual Volume Migration

A manual migration is triggered by creating a VolumeMigration resource (short name vmig) that names the PersistentVolume to move and the UUID of the destination storage node.

Migrate a single volume to a target node

kubectl apply -n simplyblock -f - <<EOF
apiVersion: storage.simplyblock.io/v1alpha1
kind: VolumeMigration
metadata:
  name: migrate-pvc-968cff4f
  namespace: simplyblock
spec:
  pvName: pvc-968cff4f-a199-4964-88f0-7cfccb5251d9
  targetNodeUUID: 4e53efdd-86c9-424f-940c-e437eb6a2e95
EOF

Both spec.pvName and spec.targetNodeUUID are immutable. To migrate the same volume again, or to a different target, create a new VolumeMigration resource.

The referenced PV must be provisioned by the Simplyblock CSI driver. The operator resolves the PV to its logical volume UUID, submits the migration to the storage API, validates the new NVMe-oF paths, and then tracks progress to completion.

Finding Target Node UUIDs

The targetNodeUUID is the backend storage node UUID, not the Kubernetes worker name.

Listing the storage node UUIDs

kubectl get storagenodeset simplyblock-node -n simplyblock \
  -o jsonpath='{.status.nodes[*].uuid}' | tr ' ' '\n'

Alternatively, the storage node CRs can be listed to find the storage node UUID:

Listing the storage node CRs

kubectl get storagenodes -n simplyblock

Monitoring a Migration

The resource exposes the current phase and snapshot progress directly in its printer columns.

Watch migration progress

kubectl get volumemigration -n simplyblock -w

Inspect full migration status

kubectl get volumemigration migrate-pvc-968cff4f \
  -n simplyblock -o jsonpath='{.status}' | jq .

Each migration progresses through the following phases, tracked in VolumeMigration.status.phase:

Phase	Description
`Pending`	The migration has been accepted. The operator is resolving the PV and submitting it.
`Validating`	The new target-side NVMe-oF paths are being established and verified by a validation Job.
`Running`	The backend is copying data and snapshots.
`Completed`	The volume now resides on the target node.
`Failed`	The migration could not complete. `status.errorMessage` holds the reason.
`Aborted`	The migration was canceled via `spec.abort`.

The status also records the resolved sourceNodeUUID, volumeUUID, poolUUID, clusterUUID, the backend migrationUUID, and startedAt / completedAt timestamps.

Aborting a Migration

An in-progress migration can be canceled by setting spec.abort to true. The phase transitions to Aborted once the backend confirms the cancellation.

Important

A volume migration can only be aborted while in the Pending or Validating phases. A running migration must be able to complete to ensure data consistency. Hence, it cannot be aborted once running.

Abort an in-progress migration

kubectl patch volumemigration migrate-pvc-968cff4f -n simplyblock \
  --type merge -p '{"spec":{"abort":true}}'

Migrating by Pinning a PVC

There are also automated processes that create a VolumeMigration resource, for example, setting the simplyblock.io/selected-storage-node annotation on an already-bound PVC. This will effectively migrate the pinned volume to a new storage node UUID. The operator create a VolumeMigration on the user's behalf, as part of moving the volume to that node. This is the same annotation that pins a volume against auto-rebalancing and node removal.

Pin a bound PVC to a new node to trigger a migration

kubectl annotate pvc <pvc-name> -n <namespace> \
  simplyblock.io/selected-storage-node=<target-storage-node-uuid> --overwrite

The annotation value must be a known storage node UUID. Any other value is rejected by a validating webhook. If the value is not a valid node, the operator records it and emits an InvalidPinTarget event.

Auto-Rebalancing

Experimental

When enabled, the operator continuously evaluates the per-node load and automatically migrates volumes off overloaded ("hot") nodes onto less-loaded ("cold") nodes. Under the hood it creates the same VolumeMigration resources as a manual migration, so all migrations remain observable through vmig.

Auto-rebalancing is configured by StorageCluster.spec.volumeAutoPlacement and is disabled by default.

Enable latency-driven auto-rebalancing

spec:
  volumeAutoPlacement:
    enabled: true
    metricsBackend: prometheus
    prometheusURL: http://prometheus.simplyblock.svc:9090
    latencyBenchmarkEnabled: true
    evaluationInterval: 60s
    imbalanceThreshold: 80
    maxVolumeMigrationsPerCycle: 10

Important

Automatic rebalancing is considered an experimental feature. Its algorithm is subject to change and is not optimal. It is not yet recommended for any production environment.

Field	Default	Description
`enabled`	`false`	Activates automatic rebalancing for the cluster.
`migrationEnabled`	`true`	When `false`, the rebalancer runs every cycle but discards the migrations instead of creating them (dry-run).
`evaluationInterval`	`60s`	How often the rebalancer evaluates load.
`imbalanceThreshold`	`80`	Minimum latency deviation from baseline (percent) before a node is considered a rebalancing source.
`minHotColdDifferencePct`	`20`	Minimum latency-deviation gap a target must be below the source before a migration is performed.
`maxVolumeMigrationsPerCycle`	`10`	Maximum number of volumes moved per cycle.
`storageNodeCandidateCount`	`3`	Number of top-loaded nodes evaluated each cycle to pick the migration source.
`defaultCoolDownSeconds`	`600`	Cool-down applied to a volume after it has been migrated, preventing it from moving again immediately.
`metricsBackend`	`prometheus`	Source of I/O metrics: `prometheus`, `controlplane`, or `uniform`.
`prometheusURL`	—	Required when `metricsBackend` is `prometheus`.
`latencyBenchmarkEnabled`	`false`	Enables `fio`-based NVMe-oF latency measurement via Kubernetes Jobs.
`latencyBenchmarkInterval`	`5m`	How often benchmark Jobs run against each storage node.
`iopsWeight`	`1.0`	Weight applied to per-volume IOPS in the volume I/O score.
`throughputWeight`	`0.1`	Weight applied to per-volume throughput (MB/s) in the volume I/O score.

Tip

Start with migrationEnabled: false (dry-run). The rebalancer still evaluates load, computes deviations, selects candidates, and emits metrics and events, but does not move any data. Once the selected candidates look correct, set migrationEnabled: true.

Note

Volumes that are pinned to a specific storage node (see Pinned Volumes) are not subject to auto-rebalancing. A one-shot placement hint from initial provisioning does not pin a volume. Such volumes remain eligible for rebalancing.

Note

Setting latencyBenchmarkEnabled: true also activates load-aware placement for newly created volumes, independent of migrationEnabled (which only controls the continuous rebalancer above). See Automatic Volume Placement.

Volume Migration During Node Draining and Removal

When a storage node is removed, the operator evacuates its volumes onto the remaining nodes before the node leaves the cluster. Removal is triggered by a StorageNodeOps resource with action: remove, and the full workflow is described in Removing a Storage Node.

Remove a storage node (drains its volumes first)

kubectl apply -n simplyblock -f - <<EOF
apiVersion: storage.simplyblock.io/v1alpha1
kind: StorageNodeOps
metadata:
  name: drain-worker-1
  namespace: simplyblock
spec:
  storageNodeRef: simplyblock-node-mejue8
  action: remove
EOF

While status.phase is Running, the removal advances through the drain sub-phases tracked in StorageNodeOps.status.subPhase:

Sub-phase	Description
`Validating`	Preconditions are checked and the node's volumes are classified.
`Suspending`	The node is suspended so no new volumes are placed on it.
`Migrating`	Volumes are migrated off the node. `status.volumesMigrated` / `status.volumesPending` track progress.
`Verifying`	Migrations are confirmed and system volumes are cleaned up.
`Removing`	The now-empty node is removed from the cluster.

System volumes are excluded from migration and deleted inline during Verifying. The set of system volumes is matched by spec.drain.systemVolumeFilterRegex (default ^sb-fio-baseline-.* which matches the auto-rebalancer volumes used to measure the system latency).

Watch a node removal drain its volumes

kubectl get storagenodeops drain-worker-1 -n simplyblock \
  -o jsonpath='{.status.subPhase} migrated={.status.volumesMigrated} pending={.status.volumesPending}{"\n"}' -w

For how removal coordinates with Kubernetes node cordon/drain and maxFaultTolerance, see Draining Coordination of a Kubernetes Worker Node.

Pinned Volumes

A volume is pinned when its PVC carries the simplyblock.io/selected-storage-node annotation. A pinned volume is never moved by auto-rebalancing. By default, a pinned volume blocks a node removal. Pinned volumes need to be explicitly directed to a target node before removal.

To allow a pinned volume to migrate during removal, set the annotation value to the UUID of the target node it should move to:

Direct a pinned volume to a specific target node before removal

kubectl annotate pvc <pvc-name> -n <namespace> \
  simplyblock.io/selected-storage-node=<target-storage-node-uuid> --overwrite

Annotation value	Removal behavior
A valid storage node UUID (different from the node being removed)	Volume is migrated to that node, removal proceeds.
Empty / absent	Volume is not pinned, a target is picked by the operator.
A non-UUID value	Removal is blocked. An `InvalidPinTarget` event is emitted.
The UUID of the node being removed	Removal is blocked. A `PinnedVolumeBlocking` event is emitted.

Volumes whose backing logical volume has no corresponding PV (for example, a volume created outside Kubernetes) also block removal, with an UnmanagedVolumeBlocking event, until they are resolved.

Check for blocking events during a stalled removal

kubectl get events -n simplyblock \
  --field-selector reason=PinnedVolumeBlocking
kubectl get events -n simplyblock \
  --field-selector reason=UnmanagedVolumeBlocking

Data Realignment

After volumes move, whether by manual migration, auto-rebalancing, or drain/removal, the operator automatically periodically re-aligns the cluster's internal data structures to the new placement so that fault-tolerance (FTT) and node-affinity guarantees are preserved. This is enabled by default and configured under volumeMigrationSettings.dataRealignment (see Enabling Volume Migration).

The operator triggers a realignment on its own schedule whenever at least one volume has moved since the last successful realignment. However, a realignment can also be trigger immediately by annotating the StorageCluster:

Trigger a data realignment immediately

kubectl annotate storagecluster simplyblock-cluster -n simplyblock \
  simplyblock.io/trigger-realignment="$(date +%s)" --overwrite

Events

The operator emits Kubernetes events on the affected resources throughout a migration. Useful reasons to filter on:

Reason	Meaning
`MigrationRequested`	A `VolumeMigration` was accepted and submitted.
`MigrationStarted`	The backend migration is running.
`MigrationCompleted`	The volume finished migrating to the target node.
`MigrationFailed`	The migration failed. See the event message and `status.errorMessage`.
`MigrationAborted`	The migration was canceled via `spec.abort`.
`MigrationStuck`	A migration has not progressed within the expected time.
`VolumeRebalancingStarted`	Auto-rebalancing began moving a volume.
`VolumeRebalancingComplete`	An auto-rebalancing migration finished.
`VolumeRebalancingDeferred`	A rebalancing move was skipped this cycle (e.g., cool-down).
`PinnedVolumeBlocking`	A pinned volume is blocking a node removal.
`UnmanagedVolumeBlocking`	A volume without a PV is blocking a node removal.
`InvalidPinTarget`	A pin annotation value is not a known storage node UUID.
`DataRealignmentTriggered`	A post-migration data realignment was started.

Stream migration-related events

kubectl get events -n simplyblock --watch \
  --field-selector reason=MigrationCompleted