Expanding a Storage Cluster

A storage cluster is expanded while it serves I/O, so no maintenance window is required. New workers are described in a growth ClusterDeploymentConfig, a deployment document that names the existing cluster in spec.clusterRef instead of describing a new one. Once the document is approved, the Simplyblock Operator creates one StorageNode per worker and slot, and every new node is integrated by the control plane on its own. The integration is followed by a rebalancing that moves data onto the new devices. The rebalancing runs at low priority, but it is still work on the data path, so an expansion is best started while the cluster is not fully utilized.

How an Expansion Runs

Every storage node a growth document creates carries spec.config.expand: true. The flag reaches the control plane with the node addition and selects the expansion path, in which the cluster keeps the status active and one expansion task is queued for the new node.

That task re-wires the role rotation of the cluster. The secondary logical volume store moves to the direct successor of the new node, and the tertiary one to the second node in line. Both donors have to be torn down and rebuilt for it, which is why the cluster reports in_expansion, and the StorageCluster the phase Provisioning, while the rotation is rebuilt. It returns to active afterward. Only then does the expansion migration start and move data onto the devices of the new node.

One expansion runs per cluster at a time. Several workers may be listed in the same document: they are integrated one after another, since the control plane refuses the addition of the next worker while an expansion is open, and the operator retries it.

Preconditions

The role rotation is only safe on a quiescent and fully redundant cluster, so the control plane refuses an expansion unless

the cluster is active and every storage node is online,
no data migration, logical volume migration, node restart, or backup task is open anywhere in the cluster, where open means anything that has not finished, and
no deletion is in flight on the two donor nodes.

The conditions are checked twice: cluster-wide when the node is added, and again with the donors of the planned role moves right before the rotation is executed.

Drafting the Growth Document

A growth document is either drafted by a discovery run or written by hand.

With a Discovery Run

An OperatorOps with the action Discover probes the workers and writes a draft document. With spec.discover.clusterRef, the draft grows the named cluster instead of describing a new one. A run that names no workers inspects every schedulable worker that does not already run a storage node.

Example of a discovery run for a growth draft (discover-growth.yaml)

apiVersion: storage.simplyblock.io/v1alpha2
kind: OperatorOps
metadata:
  name: discover-rack-c
  namespace: simplyblock
spec:
  action: Discover
  discover:
    configName: grow-rack-c
    clusterRef: simplyblock-cluster
    workers:
      - worker-7.example.com
      - worker-8.example.com

Running the discovery and finding the draft

kubectl apply -f discover-growth.yaml
kubectl get operatorops discover-rack-c -n simplyblock -w
kubectl get operatorops discover-rack-c -n simplyblock -o jsonpath='{.status.configRef}{"\n"}'

The run walks the steps Inspecting, Probing, and Writing, and it names the document it wrote in status.configRef. Without configName, the draft is named after the run. The device filters of a discovery run are described in Create a Storage Cluster.

By Hand

Example of a growth document (grow-rack-c.yaml)

apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
  name: grow-rack-c
  namespace: simplyblock
spec:
  approved: false
  clusterRef: simplyblock-cluster
  nodeSets:
    - name: rack-c
      groups:
        - name: default
          workers:
            - worker-7.example.com
            - worker-8.example.com
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]

A growth document carries no spec.cluster block. The cluster-wide settings, such as the sizing, the erasure coding scheme, and the storage-node workload, are those of the existing StorageCluster. Every group of the document uses the same device class as the cluster.

spec.cluster and spec.clusterRef decide between them what the expansion does, and a mismatch is refused rather than reconciled:

`spec.cluster.name` resolves to	`spec.clusterRef`	Expansion does
No existing `StorageCluster`	absent	Creates the cluster and all its nodes
An existing `StorageCluster`	absent	Refuses: `ClusterExists`, phase `Failed`
An existing `StorageCluster`	set to it	Adds only the nodes that do not exist yet
No existing `StorageCluster`	set	Refuses: `ClusterNotFound`, phase `Failed`

A document never removes anything. A node set left out of a later document does not drain a node, and a device removed from a group does not shrink one. Removal belongs to StorageNodeOps with action: Remove, where it is deliberate, audited, and drains first.

On a cluster with enableFailureDomains: true, every group also sets failureDomain, otherwise the new nodes are held with a FailureDomainMissing event. See Managing Failure Domains for the assignment and the balance rules.

Approving the Expansion

A draft is inert until it is approved. The validation findings of the document, for example, a device that was not found on a worker, are reported in status.message, and they are reviewed before the approval.

Reviewing the draft

kubectl get clusterdeploymentconfig grow-rack-c -n simplyblock -o yaml

Approving the expansion

kubectl patch clusterdeploymentconfig grow-rack-c -n simplyblock \
    --type=merge -p '{"spec": {"approved": true}}'

The approving edit is validated against the live cluster, and the document is immutable once it is approved. The approval cannot be withdrawn.

The document then walks the steps below and reports the phase Expanding, then Expanded.

Step	Description
`Validating`	The document is checked against the cluster and the workers.
`CreatingCluster`	Nothing is created for a growth document. The existing cluster is recorded in `status.clusterRef`.
`AwaitingCluster`	The existing cluster is confirmed.
`CreatingNodes`	One `StorageNode` per worker and slot is created, with `spec.config.expand: true`, and recorded in `status.nodeRefs`.
`Activating`	The step waits until every node the document created is `Online`.

Is an Expand Operation Needed?

No, not for nodes added by a growth document. The operator does not raise an Expand operation after a growth document, and none is needed: the nodes carry the expansion flag, so the control plane integrates each of them on its own. In the Activating step, the document raises an Activate operation named <cluster>-activate, or reuses the one of the initial deployment if it still exists. An activation of a cluster that is already active succeeds without a call to the control plane.

A StorageClusterOps with the action Expand is needed only for nodes that were added without the flag, for example, a hand-written StorageNode without spec.config.expand. Such a node follows the older path: the cluster is set to in_expansion when the node is registered, and the expansion is finalized with the operation once all new nodes are online.

Example of finalizing an expansion of nodes added without the flag

apiVersion: storage.simplyblock.io/v1alpha2
kind: StorageClusterOps
metadata:
  name: expand-simplyblock-cluster
  namespace: simplyblock
spec:
  clusterRef: simplyblock-cluster
  action: Expand

This path integrates all pending nodes at once, so it needs at least two new nodes, and at least three on a cluster with dual fault tolerance (FTT 2), to build the failover paths of every one of them. The operation runs the steps Requesting and Awaiting, and it succeeds once the cluster is active. The other actions are described in Storage Cluster Actions.

Following the Expansion

Watching the document

kubectl get clusterdeploymentconfig grow-rack-c -n simplyblock -w

Watching the new storage nodes

kubectl get storagenodes -n simplyblock -l storage.simplyblock.io/cluster=simplyblock-cluster -w

A new node walks its own provisioning steps, CheckingHost, CheckingConfig, AwaitingSlot, Posting, and Resolving, before it reaches the phase Online. How many workers are added at the same time is described in Parallel Storage Node Addition.

The cluster reports the rotation and the rebalancing that follow it. The status returns to active once the rotation is rebuilt, and rebalancing turns to false once the data has been moved.

Watching the cluster status and the rebalancing flag

kubectl get storagecluster simplyblock-cluster -n simplyblock \
    -o jsonpath='{.status.phase}{"\t"}{.status.status}{"\t"}{.status.rebalancing}{"\n"}' -w

Once the expansion is complete, the growth document can be deleted. The storage nodes it created belong to the StorageCluster and are kept.