Skip to content

Create a Storage Cluster

With the Simplyblock Operator installed and the control plane Available, the next step is to bring up a storage cluster. A storage cluster is described by a single ClusterDeploymentConfig (CDC) document. The operator writes a draft of it by probing the workers, an administrator reviews and approves it, and the operator then creates the StorageCluster, its StorageNode resources, a default storage pool, and a StorageClass.

The following overview shows the process:

Storage Cluster Lifecycle
OperatorOps (Discover)        ──► probes the workers, writes a draft
ClusterDeploymentConfig       ──► Draft
                                    │
                                    ▼  (review, then spec.approved: true)
ClusterDeploymentConfig       ──► Expanding ──► Expanded
  StorageCluster              ──► Creating ──► Online
  StoragePool <cluster>-default + StorageClass simplyblock-<namespace>-<cluster>
  StorageNode(s)              ──► Provisioning ──► Online
  StorageClusterOps <cluster>-activate
                                    │
                                    ▼  (create a PVC)
PersistentVolume              ──► Bound

Info

Not all Kubernetes workers have to become part of the simplyblock storage cluster. It is possible and common to only use a subset of all Kubernetes worker nodes for storage.

It is also possible to use a separate Kubernetes worker node pool dedicated to storage. In this case, it is important to remember to taint the nodes accordingly to prevent other workloads from being scheduled on them.

Prerequisites

OpenShift

If deploying onto an OpenShift cluster, there are additional environment-specific steps in the OpenShift guide before continuing here.

Talos

If deploying onto a Talos cluster, there are additional environment-specific steps in the Talos guide before continuing here.

EKS

If deploying onto EKS with dedicated, tainted node groups, the discovery has to tolerate the taint of the storage node group. The EKS guide shows the values and the discovery run.

GKE

On GKE with dedicated, tainted node pools, the same applies: the discovery tolerates the taint of the storage node pool, as shown in the EKS guide. The node pools are prepared as described in GKE.

Networking

Multiple ports must be open on storage node hosts. The VLAN layout and the ports are listed in Network.

The SPDK pods serve the volumes on the host network, on the pod network, or on dedicated interfaces that Multus attaches. The choice is declared in spec.cluster.network of the ClusterDeploymentConfig and cannot be changed after the cluster is created. The options, their requirements, and the production recommendations are described in Storage Network.

Discover Workers and Devices

A discovery run inspects the workers, starts one probe job per worker, and writes the result as a draft ClusterDeploymentConfig. The draft is inert: nothing is deployed until it is approved.

Automatic Initial Discovery

On a fresh installation, the operator raises the OperatorOps run initial-discovery by itself. It does so only if no OperatorOps, no ClusterDeploymentConfig, and no StorageCluster exist yet, and at least one usable worker is found. The run inspects every schedulable worker and writes the draft discovered-initial-discovery.

Follow the initial discovery
kubectl -n simplyblock get operatorops initial-discovery -w
kubectl -n simplyblock get clusterdeploymentconfig

The run moves through the steps Inspecting, Probing, and Writing, and ends in the phase Succeeded. status.configRef names the draft it wrote.

Manual Discovery

Additional discovery runs are requested with an OperatorOps resource and the action Discover. The run can be narrowed to specific workers (workers by name or nodeSelector by label, not both) and to specific devices.

discover-rack-b.yaml
apiVersion: storage.simplyblock.io/v1alpha2
kind: OperatorOps
metadata:
  name: discover-rack-b
  namespace: simplyblock
spec:
  action: Discover
  discover:
    configName: rack-b-draft
    nodeSelector:
      storage.simplyblock.io/storage: "true"
    deviceFilter:
      pcieDenyList:
        - "0000:00:1f.0"
      driveSizeRange: 1T-4T
Run the discovery
kubectl apply -f discover-rack-b.yaml
kubectl -n simplyblock get operatorops discover-rack-b -w

spec.discover narrows the run:

Field Value Purpose
configName — The draft to write. Empty names it discovered-<run name>, so a second run never overwrites a draft that may already have been reviewed.
workers Up to 128 node names The workers to inspect. Empty inspects every schedulable worker.
nodeSelector Labels The workers to inspect, by label instead of by name.
tolerations Up to 32 What the probe pods tolerate. They reach the draft as well, because the taints a run probed through are the taints the cluster has to live with.
enableControlPlaneNodes Off Also considers nodes running the API server and etcd.
deviceFilter — Narrows the devices that reach the draft, see below.
clusterRef — Writes a growth draft for an existing cluster instead of a new one.

workers and nodeSelector are mutually exclusive; a run that carries both is refused.

A probe pod is pinned to its worker with spec.nodeName, which bypasses the scheduler but not the taints. A run against a dedicated storage plane that tolerates nothing therefore inspects nothing.

Device Filters

A run scans one class of device. The PCI filters select NVMe, which is the default, and the block filters require enableLogicalBlockDevices. A filter for the class the run is not scanning is refused rather than ignored, so a narrowed run is never silently un-narrowed.

Field Class Purpose
pcieAllowList, pcieDenyList NVMe Select or exclude devices by PCI address.
pcieModel NVMe Select by model string.
enableLogicalBlockDevices Block Scans Linux block devices instead of NVMe.
blockAllowList, blockDenyList Block Select or exclude devices by path, for example, /dev/sdb.
driveSizeRange Both For example, 1T-4T.
enablePartitionedDevices Block Includes devices that already carry a partition table.

enablePartitionedDevices works with Linux block devices only, not in an NVMe setup. The Helm value discovery.enablePartitionedDevices therefore only takes effect together with discovery.enableLogicalBlockDevices. All settings that depend on the device class are listed in Settings by Device Class.

On a fleet that is uniform about which slot holds the boot device, one deny list keeps that device out of every group of every draft, which is the difference between correcting one document and correcting each of twenty groups.

The filters are inputs to the run only. The draft contains the explicit device list they produced, so re-running a filter against changed hardware cannot change what a reviewer already approved.

Review the Draft

A draft produced by discovery looks similar to the following. It lists every worker and device the run found and a proposed cluster template.

A discovered draft
apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
  name: discovered-initial-discovery
  namespace: simplyblock
spec:
  approved: false
  environment: K3s
  cluster:
    name: discovered-initial-discovery-cluster
    enableDriveFormat: true
    maxSubsystemCount: 30
    minHugePagesSize: 16G
    vcpuCount: 16
    stripe:
      dataChunks: 1
      parityChunks: 1
  nodeSets:
    - name: discovered
      groups:
        - name: group-1-nvme-4x3T
          mgmtInterface: eth0
          workers:
            - worker-01
            - worker-02
            - worker-03
          devices:
            nvme:
              - 0000:5e:00.0
              - 0000:5f:00.0
              - 0000:af:00.0
              - 0000:b0:00.0

status.message of the draft lists validation findings, for example, DeviceNotFound, StripeBelowMinimumNodes, or StripeBelowMinimumWorkers. The draft is edited in place until the findings are resolved:

Review and edit the draft
kubectl -n simplyblock get cdc discovered-initial-discovery -o wide
kubectl -n simplyblock edit cdc discovered-initial-discovery

Environment

spec.environment names the Kubernetes distribution. Discovery detects it rather than asking for it, and the value is spent once: it sets the storage node flags on the StorageCluster the document produces, after which nothing reads it again.

Value What it sets on spec.storageNodes
OpenShift The openshift block, enableCpuTopology, and enableKubeletConfiguration
Talos Disables enableKubeletConfiguration, as Talos has no writable kubelet config
Vanilla, Rancher, K3s enableKubeletConfiguration

Cluster Template

spec.cluster is the template of the storage cluster to create. It is ignored when spec.clusterRef names an existing cluster, because a growth document adds nodes to a cluster whose settings are already decided. The options and which of them can change later are explained in Cluster Deployment Options.

This is the last chance to change these

Most of the fields below are immutable on the created StorageCluster. enableDriveFormat is also destructive: it formats every device the document lists before a storage node takes it. Both are reasons the approval is a deliberate step rather than a formality.

Field Value Purpose
name Required, at most 63 characters Name of the StorageCluster. Discovery proposes <draft name>-cluster.
maxSubsystemCount Required, 10 to 500 NVMe subsystems the cluster serves.
vcpuCount Required, at least 4 vCPUs per storage node. Discovery proposes 15% of a socket, at least 5, see vCPU Count.
minHugePagesSize For example, 16G Minimum huge page memory per storage node.
stripe dataChunks and parityChunks The erasure coding scheme. Allowed: 1+0, 1+1, 2+1, 4+1, 1+2, 2+2, 4+2.
fabricType For example, tcp The NVMe-oF transport.
enableDriveFormat Off Formats every listed device. Destructive.
enableJournalDevice Off Dedicates each worker's smallest NVMe device to the journal manager.
socketsToUse, nodesPerSocket Up to 16 sockets, 1 to 8 nodes each How many storage nodes run per worker. Empty socketsToUse means socket 0 only.
nodeProvisioningBudget At least 1 How many workers are provisioned in parallel, see Parallel Storage Node Addition.
enableChecksumValidation, enableAtomicity4K Off Data integrity. enableAtomicity4K requires checksum validation.
enableFailureDomains Off Spreads data across failure domains, see below.
enableNodeAffinity Off Keeps a volume's data on the node that owns it, see Configuring Node Affinity.
kms — External KMS for volume encryption keys, see Securing the Control Plane.
backup — The S3 store backups go to, see Backup and Recovery.
ports — Overrides the NVMe-oF, RPC, and node-agent ports.
tolerations, containerResources, initContainerResources — Placement and sizing of the storage node workload.

vCPU Count

vcpuCount is the number of vCPUs every storage node of the cluster dedicates to the data plane. The value is the same for all storage nodes of a cluster, and it is fixed when the cluster is created.

When the deployment document states no vcpuCount, discovery proposes one from the smallest worker it found:

  • 15% of the logical CPUs (vCPUs) of the socket the storage node runs on, rounded down.
  • At least 5 vCPUs.
  • With several storage nodes per socket (nodesPerSocket), the 15% are shared among them.
  • Never more than the socket provides beyond the two vCPUs reserved for the kubelet and the operating system, and never less than 4.

For example, a socket with 64 vCPUs gets 9, and a socket with 320 vCPUs gets 48. The proposal suits hyper-converged workers, which share their CPUs with applications. Dedicated storage workers and I/O-intensive workloads benefit from a higher count; see Storage Node CPU for the guidance. The note in status.message of the draft states how the value was derived.

Operator images released before this fix

The operator images first released as 26.4.0-beta4, and earlier versions, propose all logical CPUs of the socket except two, for example, 318 on a socket with 320 vCPUs. On large workers, such a storage node cannot be scheduled and its addition to the cluster fails. With these images, the value must be checked and lowered in the draft before approval.

The proposal is changed in the draft before approval:

Setting the vCPU count of the draft
kubectl -n simplyblock patch cdc discovered-initial-discovery     --type=merge -p '{"spec":{"cluster":{"vcpuCount":16}}}'

After approval, the vCPU count can no longer be changed. A cluster created with a wrong value, which holds no volumes yet, is removed as described in Cleaning Up a Failed Deployment. The new discovery run writes a new draft, whose vcpuCount is corrected before approval.

Node Groups and Devices

spec.nodeSets is how the workers are grouped. A node set becomes one storage cluster, and a group within it is the set of workers that share one configuration. Workers whose hardware differs go in their own group rather than being tuned individually. Racks and zones are groups of one node set, each with its own failureDomain.

A document with a single node set creates a standalone cluster. A document with several node sets creates a supercluster, with one placement group per node set, see Superclusters below.

Level Bound
nodeSets 1 to 64
groups per set 1 to 64
workers per group 1 to 200

Each group carries:

Field Purpose
workers The Kubernetes node names.
mgmtInterface, dataInterfaces The management and data network interfaces, up to 32 data interfaces. dataInterfaces only with spdkOnHostNetwork.
devices Either nvme (PCI addresses) or block (for example, /dev/sdb), up to 128 each. Every group of a node set uses the same class.
failureDomain The failure domain of all workers in the group, see below.
spdkSystemMemory Huge page memory for the storage nodes of this group, for example, 8G.
reservedSystemCPU CPUs reserved for the host, for example, 0,1 or 0-3.
journalManager Journal copy count and per-device share.

Workers or devices that should not become part of the cluster, for example, a boot disk, are removed from the draft before approving.

A storage cluster holds at most 32 storage nodes: the workers of a node set times the storage nodes per worker. A node set above the limit is refused with MemberTooLarge.

Storage Array LUNs

On a storage array, the devices are the LUNs of each worker, listed under devices.block. Block devices require enableJournalDevice, which dedicates the smallest LUN of each worker to the journal. A discovery run with enableLogicalBlockDevices and forceJournalDevice drafts the document, or it is written by hand.

Every worker has its own LUNs with their own WWNs, so the persistent device paths differ per worker. Each worker then gets a group of its own:

Deployment document for LUNs of a storage array
apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
  name: array-cluster
  namespace: simplyblock
spec:
  approved: false
  cluster:
    name: array-cluster
    enableJournalDevice: true
    vcpuCount: 8
    minHugePagesSize: 8G
    stripe:
      dataChunks: 4
      parityChunks: 1
  nodeSets:
    - name: array
      groups:
        - name: worker-01
          workers: [worker-01]
          devices:
            block:
              - /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a01   # journal LUN, the smallest
              - /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a02   # data LUN
        - name: worker-02
          workers: [worker-02]
          devices:
            block:
              - /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a11
              - /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a12
        # ... one group per worker, at least six workers for 4+1

The LUN layout, the erasure coding scheme, and the multipathing requirements are described in Storage Arrays.

Failure Domains

With spec.cluster.enableFailureDomains: true, every group must carry a failureDomain label, for example, a rack name. The label is copied to each generated StorageNode and cannot be changed afterward.

Failure domains per group
spec:
  cluster:
    name: production
    maxSubsystemCount: 50
    vcpuCount: 8
    stripe:
      dataChunks: 2
      parityChunks: 1
    enableFailureDomains: true
  nodeSets:
    - name: racks
      groups:
        - name: rack-a
          failureDomain: rack-a
          workers: [worker-1, worker-2]
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]
        - name: rack-b
          failureDomain: rack-b
          workers: [worker-3, worker-4]
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]
        - name: rack-c
          failureDomain: rack-c
          workers: [worker-5, worker-6]
          devices:
            nvme: ["0000:01:00.0", "0000:02:00.0"]

Failure-domain mode needs at least as many failure domains as parity chunks plus two, with the same number of workers in each. A document that does not meet this is refused with FailureDomainLayout.

For details, see Failure Domains.

Superclusters

A fleet that does not fit into one storage cluster is deployed as a supercluster. Discovery drafts one on its own when the workers exceed 32 storage nodes, span several regions, or differ in network interfaces or device class. The draft then holds several node sets, and its status.message names the placement groups the approval creates.

Each node set becomes a placement group: a complete storage cluster named <cluster>-<nodeset>-<suffix>. The supercluster takes the name in spec.cluster.name, and storage pools and StorageClasses name it like any cluster. Every new volume is placed into one placement group. A node set can override the sizing and the erasure coding scheme of the template for its own placement group.

The topology, the sizing, and the validation of a supercluster are described in Superclusters and Managing Superclusters.

Copying, Splitting, and Hand-Authoring

A ClusterDeploymentConfig is an ordinary Kubernetes resource, and the discovered draft is a starting point rather than the only thing that may be approved. A reviewer can copy it under another name, split one draft into several documents that are approved separately, or write one from scratch and never run discovery at all.

Splitting is the usual reason. A draft covering three racks can become three documents, each holding one rack's groups, so each rack is reviewed and brought up on its own schedule. The first document creates the cluster and the other two name it in spec.clusterRef.

A copy is a plain kubectl get -o yaml, edited and applied under a new name. Clear metadata.resourceVersion, metadata.uid, and status from it first, as with any copied resource.

A document discovery wrote cannot be wrong about its devices, because the list came from the inspection. A hand-written one can, and the validation above is the only check a device gets before the deployment runs.

When Does the Cluster Become Active?

The erasure coding scheme determines the minimum number of storage nodes: enough to place a stripe across, which is the data chunks plus the parity chunks, plus one spare per tolerated failure to rebuild onto.

Scheme Minimum storage nodes
1+0 1
1+1 3
2+1 4
1+2 5
4+1 6
2+2 6
4+2 8

The approval is refused if the document provides fewer nodes, with StripeBelowMinimumNodes, or if they sit on too few workers, with StripeBelowMinimumWorkers. The second is counted separately because a fleet that reaches the number by running several nodes per socket on two workers has not bought the independent spare the count asks for: every node of a worker fails with the worker. An unstated scheme counts as 1+1.

The operator activates the cluster as soon as all storage nodes of the document are Online. In a supercluster, the minimum applies to each node set, and each placement group is activated once its own storage nodes are Online.

Approve the Deployment

Once the draft is correct, it is approved by setting spec.approved: true. The admission webhook validates the approving edit against the live cluster, for example, that the devices exist. After approval, the document is immutable and the approval cannot be withdrawn.

Approve the draft
kubectl -n simplyblock patch cdc discovered-initial-discovery \
    --type=merge -p '{"spec":{"approved":true}}'

kubectl -n simplyblock get cdc discovered-initial-discovery -w

The document moves from Draft to Expanding and finally to Expanded (or Failed). While expanding, it passes the steps Validating, CreatingCluster, AwaitingCluster, CreatingNodes, and Activating.

Warning

Simplyblock exclusively owns the resources it has been allocated. It must be ensured they are sized correctly alongside other workloads.

Additionally, simplyblock manages huge page allocation automatically. Total RAM required depends on vCPU count, the number of active logical volumes, and utilized virtual storage per node.

More information can be found in Minimum Hardware Requirements.

What Gets Created

The expansion creates the following resources:

  • StorageCluster: Named after spec.cluster.name, in the namespace of the document. For a supercluster, one further StorageCluster is created per node set as its placement group.
  • StoragePool <cluster>-default: The default pool. It is created once and never recreated.
  • StorageClass simplyblock-<namespace>-<cluster>: The class of the default pool, with the provisioner csi.simplyblock.io, WaitForFirstConsumer binding, the reclaim policy Delete, and volume expansion enabled. It is not marked as the Kubernetes default class.
  • StorageNode resources: One per worker, socket, and node per socket. Each is provisioned by the storage node DaemonSet, and its devices appear as StorageDevice resources.
  • StorageClusterOps <cluster>-activate: Raised once all storage nodes are Online.

With a cluster named production in the namespace simplyblock, the default pool is production-default and the StorageClass is simplyblock-simplyblock-production.

The ClusterDeploymentConfig is no longer needed after it reaches Expanded and can be deleted.

Verify the Cluster

Check the storage cluster and its resources
kubectl -n simplyblock get storagecluster,storagenode,storagedevice,storagepool

The same resources can be listed with their short names, kubectl -n simplyblock get stc,sn,sd,sp.

Example output of kubectl -n simplyblock get stc
NAME         PHASE    ROLE         STEP   STATUS   EC    UUID                                   AGE
production   Online   Standalone          active   2x1   bfa260ce-06a7-4bcb-a843-813d0be633af   12m

The StorageCluster phase is Online once the cluster is active. Degraded and Unavailable indicate problems, and status.message explains them. The storage nodes report Online in their phase column.

The operator also writes the cluster credentials for the CSI driver into the Secret simplyblock-csi-secret-v2 in the operator's namespace. The Secret is maintained by the operator and does not need to be managed manually.

Cleaning Up a Failed Deployment

A deployment that never reached Online (for example, because a storage node could not be added) is cleaned up in one of two ways:

  • Resource cleanup removes only the storage cluster and its documents. The operator, the control plane, FoundationDB, the CSI driver, and the certificates stay in place, and discovery and deployment are run again. It is the faster way and is tried first.
  • Full cleanup removes the whole installation, including the operator, and is followed by a new installation. It is used when the resource cleanup does not complete, or when the installation itself is broken.

Both apply only to a cluster that holds no volumes yet.

The reason of the failure is found and fixed first, otherwise the next attempt fails the same way. Typical signs are a StorageCluster that stays in the phase Provisioning, a node_add task with the status suspended, and NodeAddFailing events on the storage nodes:

Finding the reason of a failed deployment
kubectl -n simplyblock get storagecluster -o yaml | grep -A12 "^  status:"
kubectl -n simplyblock get events --field-selector reason=NodeAddFailing
kubectl -n simplyblock get events --field-selector reason=FailedScheduling

Resource Cleanup for Rerunning Discovery and Deployment

The resources are removed in order, each one after the previous is gone. Deleting the StorageCluster also deletes the cluster in the control plane:

Removing the resources of a failed deployment
NS=simplyblock

kubectl -n $NS delete storagenodes --all --wait=true --timeout=10m
kubectl -n $NS delete storagenodesets --all --wait=true --timeout=10m
kubectl -n $NS delete storagecluster --all --wait=true --timeout=10m
kubectl -n $NS delete clusterdeploymentconfigs --all --wait=true
kubectl -n $NS delete operatorops --all --wait=true

kubectl -n $NS get storagecluster,storagenodes,storagenodesets,clusterdeploymentconfigs,operatorops

On OpenShift, the machine configuration of the failed storage nodes is removed as well (see Machine Configuration of a Failed Deployment).

With no StorageCluster, ClusterDeploymentConfig, or OperatorOps left, the operator starts the initial discovery again and writes a new draft. The draft is corrected (for example, its vCPU count) and approved as described above.

A resource that stays in Terminating after the timeout has a finalizer the operator cannot clear, typically because the control plane refused to delete the cluster or a half-added storage node. The operator log names the reason. If it cannot be resolved, the full cleanup is used instead.

Full Cleanup for Redeployment

The whole installation is removed with the cleanup script of the Simplyblock Operator (see Removing a Previous Installation), which deletes the custom resources, their finalizers, the operator, node labels, and persistent volumes:

Removing the whole installation
curl -sO https://raw.githubusercontent.com/simplyblock/simplyblock-operator/main/helm-charts/scripts/cleanup-simplyblock.sh
chmod +x cleanup-simplyblock.sh

export KUBECONFIG=/path/to/kubeconfig
./cleanup-simplyblock.sh simplyblock

The script does not remove machine configuration on OpenShift, which is removed as described in Machine Configuration of a Failed Deployment.

The operator is then installed again (see Installing the Operator). The initial discovery runs as on a new installation and writes a new draft, which is corrected and approved.

Create a Storage Pool

Volumes can be provisioned from the default pool right away. Additional pools divide the cluster into tenancy units with their own capacity and QoS limits. A StoragePool references its cluster with clusterRef.

storage-pool.yaml
apiVersion: storage.simplyblock.io/v1alpha2
kind: StoragePool
metadata:
  name: tenant-a
  namespace: simplyblock
spec:
  clusterRef: production
  limits:
    capacity: 10T
    maxVolumeSize: 2T
    iops: 200000
    throughput:
      readWrite: 4096
  volumeDefaults:
    iops: 20000
    throughput:
      readWrite: 512
    filesystem: xfs
Create the pool
kubectl apply -f storage-pool.yaml
kubectl -n simplyblock get storagepool tenant-a

spec.limits (capacity, maximum volume size, IOPS, and throughput in MB/s) can be changed later. spec.volumeDefaults is immutable once set. spec.allowedNodes restricts the pool to specific Kubernetes nodes, see Host Authentication and Encryption.

Deleting a Pool

A pool carries the finalizer storage.simplyblock.io/storagepool-finalizer and refuses to finish deleting while anything Kubernetes knows about still refers to it.

Deleting a pool with Result
An authored StorageClass assigned to it Held. StorageClassStillAssigned, requeued, nothing deleted
Only the operator's own default class The class is deleted, then the backend pool, then the finalizer clears
A PersistentVolume in it Held. VolumesStillBound, requeued, nothing deleted
Neither The backend pool is deleted and the finalizer clears

An authored class holds the deletion because the operator neither wrote it nor knows why it exists, and deleting somebody's provisioning contract to let a pool go is not a trade it makes. Deleting the class releases the pool. The operator's own default class does not hold, because removing it is cleanup rather than a decision.

A StorageCluster owns its pools, so deleting one cascades, and is held behind any pool that is itself held.

Author a StorageClass for the Pool

The operator creates a StorageClass only for the default pool. For every other pool, the StorageClass is written by the administrator and linked to the pool with three labels: storage.simplyblock.io/namespace, storage.simplyblock.io/cluster, and storage.simplyblock.io/pool. The parameters cluster_id (the cluster's status.uuid) and pool_name direct the CSI driver to the pool.

storage-class-tenant-a.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tenant-a-fast
  labels:
    storage.simplyblock.io/namespace: simplyblock
    storage.simplyblock.io/cluster: production
    storage.simplyblock.io/pool: tenant-a
provisioner: csi.simplyblock.io
parameters:
  cluster_id: "<StorageCluster status.uuid>"
  pool_name: tenant-a
  max_iops: "20000"
  csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The pool lists its assigned classes in status.storageClassNames. For the full parameter mapping, see Storage Class.

Provision the First Volume

Now, everything is in place to create the first volume. The operator has deployed the simplyblock CSI driver into the Kubernetes cluster. Hence, creating a volume is as simple as creating a PersistentVolumeClaim with the correct StorageClass set.

Create the PVC

test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: simplyblock-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: simplyblock-simplyblock-production
Create the PVC
kubectl apply -f test-pvc.yaml
kubectl get pvc simplyblock-test-pvc
Example output of PVC status
NAME                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                         AGE
simplyblock-test-pvc    Pending                                       simplyblock-simplyblock-production   5s

Since provisioning is asynchronous, the PVC status is initially Pending. The StorageClass uses WaitForFirstConsumer, which means the volume is not provisioned until a pod actually needs it. The scheduler picks the right node first, then the volume is created close to where it will be used.

Mount the Volume into a Test Pod

To mount the volume, it can be used like any other Kubernetes persistent volume claim by referencing it in the pod's volumes specification.

test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: simplyblock-test-pod
spec:
  containers:
    - name: test
      image: busybox
      command: ["/bin/sh", "-c", "echo 'volume provisioned successfully' > /data/test.txt && sleep 3600"]
      volumeMounts:
        - mountPath: /data
          name: storage
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: simplyblock-test-pvc
Create the pod
kubectl apply -f test-pod.yaml

When the pod reaches Running status, the PVC changes to bound.

Check the PVC status
kubectl get pvc simplyblock-test-pvc
Example output of PVC status
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                         AGE
simplyblock-test-pvc    Bound    pvc-3f2a1c9e-84b1-4d2e-9f3a-1234abcd5678   10Gi       RWO            simplyblock-simplyblock-production   30s

It is now possible to access the data written to the volume when the pod started up.

Check the volume contents
kubectl exec simplyblock-test-pod -- cat /data/test.txt
Example output of volume contents
volume provisioned successfully

The cluster is now fully operational and the test resources should be cleaned up with:

Cleanup the test resources
kubectl delete pod simplyblock-test-pod
kubectl delete pvc simplyblock-test-pvc

Next Steps

The cluster serves volumes now. Two things it does not yet have are more capacity and a second cluster, and both are their own deployment document.

More nodes for this cluster. A ClusterDeploymentConfig that names the cluster in spec.clusterRef, and carries no spec.cluster block, adds nodes to it. A discovery run with spec.discover.clusterRef writes such a growth draft from whatever workers are still unclaimed. See Expanding a Storage Cluster.

A second storage cluster. Several storage clusters can share a namespace, and each one is pointed at a different set of workers. Clusters that are meant to act as one are deployed as a supercluster instead, see Managing Superclusters.

From here, Operations covers running the cluster, and Usage covers provisioning volumes from it.