Create a Storage Cluster

With the Simplyblock Operator installed and the control plane Available, the next step is to bring up a storage cluster. A storage cluster is described by a single ClusterDeploymentConfig (CDC) document. The operator writes a draft of it by probing the workers, an administrator reviews and approves it, and the operator then creates the StorageCluster, its StorageNode resources, a default storage pool, and a StorageClass.

The following overview shows the process:

Storage Cluster Lifecycle

OperatorOps (Discover)        ──► probes the workers, writes a draft
ClusterDeploymentConfig       ──► Draft
                                    │
                                    ▼  (review, then spec.approved: true)
ClusterDeploymentConfig       ──► Expanding ──► Expanded
  StorageCluster              ──► Creating ──► Online
  StoragePool <cluster>-default + StorageClass simplyblock-<namespace>-<cluster>
  StorageNode(s)              ──► Provisioning ──► Online
  StorageClusterOps <cluster>-activate
                                    │
                                    ▼  (create a PVC)
PersistentVolume              ──► Bound

Info

Not all Kubernetes workers have to become part of the simplyblock storage cluster. It is possible and common to only use a subset of all Kubernetes worker nodes for storage.

It is also possible to use a separate Kubernetes worker node pool dedicated to storage. In this case, it is important to remember to taint the nodes accordingly to prevent other workloads from being scheduled on them.

Prerequisites

OpenShift

If deploying onto an OpenShift cluster, there are additional environment-specific steps in the OpenShift guide before continuing here.

Talos

If deploying onto a Talos cluster, there are additional environment-specific steps in the Talos guide before continuing here.

Networking

Multiple ports must be open on storage node hosts.

It is required to use one or more separate VLANs for simplyblock. Ports within the same VLAN do not require extra firewall rules, but ports between the control plane and storage networks typically do.

Service	Direction	Hosts	Network	Port(s)	Protocol(s)
ICMP	egress	control	Control	-	ICMP
storage-node-api	egress	storage	Control	5000	TCP
NVMf (client-target)	egress	client	Storage	4420-4499	TCP, UDP ⁽¹⁾
FoundationDB	ingress, egress	control	Control	4500	TCP
control-plane-API	ingress	control	Control	80	TCP
storage-node-RPC	ingress, egress	storage, control	Control	8080-9044	TCP
Monitoring Stack ⁽²⁾	ingress, egress	control	Control	12202, 13301, 13302, 9200, 9090	TCP

¹ The NVMf port range carries both transports. NVMe-oF/TCP uses it over TCP, NVMe-oF/RDMA over UDP. Only the protocol of the transport(s) in use has to be opened.
² The monitoring stack is optional. Its ports only have to be opened where it is deployed.

Discover Workers and Devices

A discovery run inspects the workers, starts one probe job per worker, and writes the result as a draft ClusterDeploymentConfig. The draft is inert: nothing is deployed until it is approved.

Automatic Initial Discovery

On a fresh installation, the operator raises the OperatorOps run initial-discovery by itself. It does so only if no OperatorOps, no ClusterDeploymentConfig, and no StorageCluster exist yet, and at least one usable worker is found. The run inspects every schedulable worker and writes the draft discovered-initial-discovery.

Follow the initial discovery

kubectl -n simplyblock get operatorops initial-discovery -w
kubectl -n simplyblock get clusterdeploymentconfig

The run moves through the steps Inspecting, Probing, and Writing, and ends in the phase Succeeded. status.configRef names the draft it wrote.

Manual Discovery

Additional discovery runs are requested with an OperatorOps resource and the action Discover. The run can be narrowed to specific workers (workers by name or nodeSelector by label, not both) and to specific devices.

discover-rack-b.yaml

apiVersion: storage.simplyblock.io/v1alpha2
kind: OperatorOps
metadata:
  name: discover-rack-b
  namespace: simplyblock
spec:
  action: Discover
  discover:
    configName: rack-b-draft
    nodeSelector:
      storage.simplyblock.io/storage: "true"
    deviceFilter:
      pcieDenyList:
        - "0000:00:1f.0"
      driveSizeRange: 1T-4T
      enablePartitionedDevices: true

Run the discovery

kubectl apply -f discover-rack-b.yaml
kubectl -n simplyblock get operatorops discover-rack-b -w

spec.discover narrows the run:

Field	Value	Purpose
`configName`	—	The draft to write. Empty names it `discovered-<run name>`, so a second run never overwrites a draft that may already have been reviewed.
`workers`	Up to 128 node names	The workers to inspect. Empty inspects every schedulable worker.
`nodeSelector`	Labels	The workers to inspect, by label instead of by name.
`tolerations`	Up to 32	What the probe pods tolerate. They reach the draft as well, because the taints a run probed through are the taints the cluster has to live with.
`enableControlPlaneNodes`	Off	Also considers nodes running the API server and etcd.
`deviceFilter`	—	Narrows the devices that reach the draft, see below.
`clusterRef`	—	Writes a growth draft for an existing cluster instead of a new one.

workers and nodeSelector are mutually exclusive; a run that carries both is refused.

A probe pod is pinned to its worker with spec.nodeName, which bypasses the scheduler but not the taints. A run against a dedicated storage plane that tolerates nothing therefore inspects nothing.

Device Filters

A run scans one class of device. The PCI filters select NVMe, which is the default, and the block filters require enableLogicalBlockDevices. A filter for the class the run is not scanning is refused rather than ignored, so a narrowed run is never silently un-narrowed.

Field	Class	Purpose
`pcieAllowList`, `pcieDenyList`	NVMe	Select or exclude devices by PCI address.
`pcieModel`	NVMe	Select by model string.
`enableLogicalBlockDevices`	Block	Scans Linux block devices instead of NVMe.
`blockAllowList`, `blockDenyList`	Block	Select or exclude devices by path, for example, `/dev/sdb`.
`driveSizeRange`	Both	For example, `1T-4T`.
`enablePartitionedDevices`	Both	Includes devices that already carry a partition table.

On a fleet that is uniform about which slot holds the boot device, one deny list keeps that device out of every group of every draft, which is the difference between correcting one document and correcting each of twenty groups.

The filters are inputs to the run only. The draft contains the explicit device list they produced, so re-running a filter against changed hardware cannot change what a reviewer already approved.

Review the Draft

A draft produced by discovery looks similar to the following. It lists every worker and device the run found and a proposed cluster template.

A discovered draft

apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
  name: discovered-initial-discovery
  namespace: simplyblock
spec:
  approved: false
  environment: K3s
  cluster:
    name: discovered-initial-discovery-cluster
    enableDriveFormat: true
    maxSubsystemCount: 30
    minHugePagesSize: 16G
    vcpuCount: 16
    stripe:
      dataChunks: 1
      parityChunks: 1
  nodeSets:
    - name: discovered
      groups:
        - name: group-1-nvme-4x3T
          mgmtInterface: eth0
          workers:
            - worker-01
            - worker-02
            - worker-03
          devices:
            nvme:
              - 0000:5e:00.0
              - 0000:5f:00.0
              - 0000:af:00.0
              - 0000:b0:00.0

status.message of the draft lists validation findings, for example, DeviceNotFound, StripeBelowMinimumNodes, or StripeBelowMinimumWorkers. The draft is edited in place until the findings are resolved:

Review and edit the draft

kubectl -n simplyblock get cdc discovered-initial-discovery -o wide
kubectl -n simplyblock edit cdc discovered-initial-discovery

Environment

spec.environment names the Kubernetes distribution. Discovery detects it rather than asking for it, and the value is spent once: it sets the storage node flags on the StorageCluster the document produces, after which nothing reads it again.

Value	What it sets on `spec.storageNodes`
`OpenShift`	The `openshift` block, `enableCpuTopology`, and `enableKubeletConfiguration`
`Talos`	Disables `enableKubeletConfiguration`, as Talos has no writable kubelet config
`Vanilla`, `Rancher`, `K3s`	`enableKubeletConfiguration`

Cluster Template

spec.cluster is the template of the storage cluster to create. It is ignored when spec.clusterRef names an existing cluster, because a growth document adds nodes to a cluster whose settings are already decided.

This is the last chance to change these

Most of the fields below are immutable on the created StorageCluster. enableDriveFormat is also destructive: it formats every device the document lists before a storage node takes it. Both are reasons the approval is a deliberate step rather than a formality.

Field	Value	Purpose
`name`	Required, at most 63 characters	Name of the `StorageCluster`. Discovery proposes `<draft name>-cluster`.
`maxSubsystemCount`	Required, 10 to 75	NVMe subsystems the cluster serves.
`vcpuCount`	Required, at least 4	vCPUs per storage node.
`minHugePagesSize`	For example, `16G`	Minimum huge page memory per storage node.
`stripe`	`dataChunks` and `parityChunks`	The erasure coding scheme. Allowed: 1+0, 1+1, 2+1, 4+1, 1+2, 2+2, 4+2.
`fabricType`	For example, `tcp`	The NVMe-oF transport.
`enableDriveFormat`	Off	Formats every listed device. Destructive.
`enableJournalDevice`	Off	Dedicates each worker's smallest NVMe device to the journal manager.
`socketsToUse`, `nodesPerSocket`	Up to 16 sockets, 1 to 8 nodes each	How many storage nodes run per worker. Empty `socketsToUse` means socket 0 only.
`nodeProvisioningBudget`	At least 1	How many workers are provisioned in parallel, see Parallel Storage Node Addition.
`enableChecksumValidation`, `enableAtomicity4K`	Off	Data integrity. `enableAtomicity4K` requires checksum validation.
`enableFailureDomains`	Off	Spreads data across failure domains, see below.
`enableNodeAffinity`	Off	Keeps a volume's data on the node that owns it, see Configuring Node Affinity.
`kms`	—	External KMS for volume encryption keys, see Securing the Control Plane.
`backup`	—	The S3 store backups go to, see Backup and Recovery.
`ports`	—	Overrides the NVMe-oF, RPC, and node-agent ports.
`tolerations`, `containerResources`, `initContainerResources`	—	Placement and sizing of the storage node workload.

Node Groups and Devices

spec.nodeSets is how the workers are grouped. A set is an organizational unit, usually a rack; a group within it is the set of workers that share one configuration. Workers whose hardware differs go in their own group rather than being tuned individually.

Level	Bound
`nodeSets`	1 to 64
`groups` per set	1 to 64
`workers` per group	1 to 200

Each group carries:

Field	Purpose
`workers`	The Kubernetes node names.
`mgmtInterface`, `dataInterfaces`	The management and data network interfaces, up to 32 data interfaces.
`devices`	Either `nvme` (PCI addresses) or `block` (for example, `/dev/sdb`), up to 128 each. Every group of a document uses the same class.
`failureDomain`	The failure domain of all workers in the group, see below.
`spdkSystemMemory`	Huge page memory for the storage nodes of this group, for example, `8G`.
`reservedSystemCPU`	CPUs reserved for the host, for example, `0,1` or `0-3`.
`journalManager`	Journal copy count and per-device share.

Workers or devices that should not become part of the cluster, for example, a boot disk, are removed from the draft before approving.

Failure Domains

With spec.cluster.enableFailureDomains: true, every group must carry a failureDomain label, for example, a rack name. The label is copied to each generated StorageNode and cannot be changed afterward.

Failure domains per group

spec:
  cluster:
    name: production
    maxSubsystemCount: 50
    vcpuCount: 8
    stripe:
      dataChunks: 2
      parityChunks: 1
    enableFailureDomains: true
  nodeSets:
    - name: racks
      groups:
        - name: rack-a
          failureDomain: rack-a
          workers: [worker-1, worker-2]
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]
        - name: rack-b
          failureDomain: rack-b
          workers: [worker-3, worker-4]
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]

For details, see Failure Domains.

Copying, Splitting, and Hand-Authoring

A ClusterDeploymentConfig is an ordinary Kubernetes resource, and the discovered draft is a starting point rather than the only thing that may be approved. A reviewer can copy it under another name, split one draft into several documents that are approved separately, or write one from scratch and never run discovery at all.

Splitting is the usual reason. A draft covering three racks can become three documents, each holding one rack's node set, so each rack is reviewed and brought up on its own schedule. The first document creates the cluster and the other two name it in spec.clusterRef.

A copy is a plain kubectl get -o yaml, edited and applied under a new name. Clear metadata.resourceVersion, metadata.uid, and status from it first, as with any copied resource.

A document discovery wrote cannot be wrong about its devices, because the list came from the inspection. A hand-written one can, and the validation above is the only check a device gets before the deployment runs.

When Does the Cluster Become Active?

The erasure coding scheme determines the minimum number of storage nodes: enough to place a stripe across, which is the data chunks plus the parity chunks, plus one spare per tolerated failure to rebuild onto.

Scheme	Minimum storage nodes
1+0	1
1+1	3
2+1	4
1+2	5
4+1	6
2+2	6
4+2	8

The approval is refused if the document provides fewer nodes, with StripeBelowMinimumNodes, or if they sit on too few workers, with StripeBelowMinimumWorkers. The second is counted separately because a fleet that reaches the number by running several nodes per socket on two workers has not bought the independent spare the count asks for: every node of a worker fails with the worker. An unstated scheme counts as 1+1.

The operator activates the cluster as soon as all storage nodes of the document are Online.

Approve the Deployment

Once the draft is correct, it is approved by setting spec.approved: true. The admission webhook validates the approving edit against the live cluster, for example, that the devices exist. After approval, the document is immutable and the approval cannot be withdrawn.

Approve the draft

kubectl -n simplyblock patch cdc discovered-initial-discovery \
    --type=merge -p '{"spec":{"approved":true}}'

kubectl -n simplyblock get cdc discovered-initial-discovery -w

The document moves from Draft to Expanding and finally to Expanded (or Failed). While expanding, it passes the steps Validating, CreatingCluster, AwaitingCluster, CreatingNodes, and Activating.

Warning

Simplyblock exclusively owns the resources it has been allocated. It must be ensured they are sized correctly alongside other workloads.

Additionally, simplyblock manages huge page allocation automatically. Total RAM required depends on vCPU count, the number of active logical volumes, and utilized virtual storage per node.

More information can be found in Minimum Hardware Requirements.

What Gets Created

The expansion creates the following resources:

StorageCluster: Named after spec.cluster.name, in the namespace of the document. A namespace holds at most one storage cluster.
StoragePool <cluster>-default: The default pool. It is created once and never recreated.
StorageClass simplyblock-<namespace>-<cluster>: The class of the default pool, with the provisioner csi.simplyblock.io, WaitForFirstConsumer binding, the reclaim policy Delete, and volume expansion enabled. It is not marked as the Kubernetes default class.
StorageNode resources: One per worker, socket, and node per socket. Each is provisioned by the storage node DaemonSet, and its devices appear as StorageDevice resources.
StorageClusterOps <cluster>-activate: Raised once all storage nodes are Online.

With a cluster named production in the namespace simplyblock, the default pool is production-default and the StorageClass is simplyblock-simplyblock-production.

The ClusterDeploymentConfig is no longer needed after it reaches Expanded and can be deleted.

Verify the Cluster

Check the storage cluster and its resources

kubectl -n simplyblock get storagecluster,storagenode,storagedevice,storagepool

The same resources can be listed with their short names, kubectl -n simplyblock get stc,sn,sd,sp.

Example output of kubectl -n simplyblock get stc

NAME         PHASE    STEP   STATUS   EC    UUID                                   AGE
production   Online          active   2x1   bfa260ce-06a7-4bcb-a843-813d0be633af   12m

The StorageCluster phase is Online once the cluster is active. Degraded and Unavailable indicate problems, and status.message explains them. The storage nodes report Online in their phase column.

The operator also writes the cluster credentials for the CSI driver into the Secret simplyblock-csi-secret-v2 in the operator's namespace. The Secret is maintained by the operator and does not need to be managed manually.

Create a Storage Pool

Volumes can be provisioned from the default pool right away. Additional pools divide the cluster into tenancy units with their own capacity and QoS limits. A StoragePool references its cluster with clusterRef.

storage-pool.yaml

apiVersion: storage.simplyblock.io/v1alpha2
kind: StoragePool
metadata:
  name: tenant-a
  namespace: simplyblock
spec:
  clusterRef: production
  limits:
    capacity: 10T
    maxVolumeSize: 2T
    iops: 200000
    throughput:
      readWrite: 4096
  volumeDefaults:
    iops: 20000
    throughput:
      readWrite: 512
    filesystem: xfs

Create the pool

kubectl apply -f storage-pool.yaml
kubectl -n simplyblock get storagepool tenant-a

spec.limits (capacity, maximum volume size, IOPS, and throughput in MB/s) can be changed later. spec.volumeDefaults is immutable once set. spec.allowedNodes restricts the pool to specific Kubernetes nodes, see Host Authentication and Encryption.

Deleting a Pool

A pool carries the finalizer storage.simplyblock.io/storagepool-finalizer and refuses to finish deleting while anything Kubernetes knows about still refers to it.

Deleting a pool with	Result
An authored `StorageClass` assigned to it	Held. `StorageClassStillAssigned`, requeued, nothing deleted
Only the operator's own default class	The class is deleted, then the backend pool, then the finalizer clears
A `PersistentVolume` in it	Held. `VolumesStillBound`, requeued, nothing deleted
Neither	The backend pool is deleted and the finalizer clears

An authored class holds the deletion because the operator neither wrote it nor knows why it exists, and deleting somebody's provisioning contract to let a pool go is not a trade it makes. Deleting the class releases the pool. The operator's own default class does not hold, because removing it is cleanup rather than a decision.

A StorageCluster owns its pools, so deleting one cascades, and is held behind any pool that is itself held.

Author a StorageClass for the Pool

The operator creates a StorageClass only for the default pool. For every other pool, the StorageClass is written by the administrator and linked to the pool with three labels: storage.simplyblock.io/namespace, storage.simplyblock.io/cluster, and storage.simplyblock.io/pool. The parameters cluster_id (the cluster's status.uuid) and pool_name direct the CSI driver to the pool.

storage-class-tenant-a.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tenant-a-fast
  labels:
    storage.simplyblock.io/namespace: simplyblock
    storage.simplyblock.io/cluster: production
    storage.simplyblock.io/pool: tenant-a
provisioner: csi.simplyblock.io
parameters:
  cluster_id: "<StorageCluster status.uuid>"
  pool_name: tenant-a
  max_iops: "20000"
  csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The pool lists its assigned classes in status.storageClassNames. For the full parameter mapping, see Storage Class.

Provision the First Volume

Now, everything is in place to create the first volume. The operator has deployed the simplyblock CSI driver into the Kubernetes cluster. Hence, creating a volume is as simple as creating a PersistentVolumeClaim with the correct StorageClass set.

Create the PVC

test-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: simplyblock-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: simplyblock-simplyblock-production

Create the PVC

kubectl apply -f test-pvc.yaml
kubectl get pvc simplyblock-test-pvc

Example output of PVC status

NAME                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                         AGE
simplyblock-test-pvc    Pending                                       simplyblock-simplyblock-production   5s

Since provisioning is asynchronous, the PVC status is initially Pending. The StorageClass uses WaitForFirstConsumer, which means the volume is not provisioned until a pod actually needs it. The scheduler picks the right node first, then the volume is created close to where it will be used.

Mount the Volume into a Test Pod

To mount the volume, it can be used like any other Kubernetes persistent volume claim by referencing it in the pod's volumes specification.

test-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: simplyblock-test-pod
spec:
  containers:
    - name: test
      image: busybox
      command: ["/bin/sh", "-c", "echo 'volume provisioned successfully' > /data/test.txt && sleep 3600"]
      volumeMounts:
        - mountPath: /data
          name: storage
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: simplyblock-test-pvc

Create the pod

kubectl apply -f test-pod.yaml

When the pod reaches Running status, the PVC changes to bound.

Check the PVC status

kubectl get pvc simplyblock-test-pvc

Example output of PVC status

NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                         AGE
simplyblock-test-pvc    Bound    pvc-3f2a1c9e-84b1-4d2e-9f3a-1234abcd5678   10Gi       RWO            simplyblock-simplyblock-production   30s

It is now possible to access the data written to the volume when the pod started up.

Check the volume contents

kubectl exec simplyblock-test-pod -- cat /data/test.txt

Example output of volume contents

volume provisioned successfully

The cluster is now fully operational and the test resources should be cleaned up with:

Cleanup the test resources

kubectl delete pod simplyblock-test-pod
kubectl delete pvc simplyblock-test-pvc

Next Steps

The cluster serves volumes now. Two things it does not yet have are more capacity and a second cluster, and both are their own deployment document.

More nodes for this cluster. A ClusterDeploymentConfig that names the cluster in spec.clusterRef, and carries no spec.cluster block, adds nodes to it. A discovery run with spec.discover.clusterRef writes such a growth draft from whatever workers are still unclaimed. See Expanding a Storage Cluster.

A second storage cluster. A namespace holds at most one StorageCluster, so another cluster needs a namespace of its own, and it is recommended to point each one at a different set of workers.

From here, Operations covers running the cluster, and Usage covers provisioning volumes from it.