Create a Storage Cluster
With the Simplyblock Operator installed and the control plane Available, the next step is
to bring up a storage cluster. A storage cluster is described by a single ClusterDeploymentConfig (CDC) document.
The operator writes a draft of it by probing the workers, an administrator reviews and approves it, and the operator
then creates the StorageCluster, its StorageNode resources, a default storage pool, and a StorageClass.
The following overview shows the process:
OperatorOps (Discover) ──► probes the workers, writes a draft
ClusterDeploymentConfig ──► Draft
│
▼ (review, then spec.approved: true)
ClusterDeploymentConfig ──► Expanding ──► Expanded
StorageCluster ──► Creating ──► Online
StoragePool <cluster>-default + StorageClass simplyblock-<namespace>-<cluster>
StorageNode(s) ──► Provisioning ──► Online
StorageClusterOps <cluster>-activate
│
▼ (create a PVC)
PersistentVolume ──► Bound
Info
Not all Kubernetes workers have to become part of the simplyblock storage cluster. It is possible and common to only use a subset of all Kubernetes worker nodes for storage.
It is also possible to use a separate Kubernetes worker node pool dedicated to storage. In this case, it is important to remember to taint the nodes accordingly to prevent other workloads from being scheduled on them.
Prerequisites
OpenShift
If deploying onto an OpenShift cluster, there are additional environment-specific steps in the OpenShift guide before continuing here.
Talos
If deploying onto a Talos cluster, there are additional environment-specific steps in the Talos guide before continuing here.
EKS
If deploying onto EKS with dedicated, tainted node groups, the discovery has to tolerate the taint of the storage node group. The EKS guide shows the values and the discovery run.
GKE
On GKE with dedicated, tainted node pools, the same applies: the discovery tolerates the taint of the storage node pool, as shown in the EKS guide. The node pools are prepared as described in GKE.
Networking
Multiple ports must be open on storage node hosts. The VLAN layout and the ports are listed in Network.
The SPDK pods serve the volumes on the host network, on the pod network, or on dedicated interfaces that Multus
attaches. The choice is declared in spec.cluster.network of the ClusterDeploymentConfig and cannot be changed after
the cluster is created. The options, their requirements, and the production recommendations are described in
Storage Network.
Discover Workers and Devices
A discovery run inspects the workers, starts one probe job per worker, and writes the result as a draft
ClusterDeploymentConfig. The draft is inert: nothing is deployed until it is approved.
Automatic Initial Discovery
On a fresh installation, the operator raises the OperatorOps run initial-discovery by itself. It does so only if
no OperatorOps, no ClusterDeploymentConfig, and no StorageCluster exist yet, and at least one usable worker is
found. The run inspects every schedulable worker and writes the draft discovered-initial-discovery.
kubectl -n simplyblock get operatorops initial-discovery -w
kubectl -n simplyblock get clusterdeploymentconfig
The run moves through the steps Inspecting, Probing, and Writing, and ends in the phase Succeeded.
status.configRef names the draft it wrote.
Manual Discovery
Additional discovery runs are requested with an OperatorOps resource and the action Discover. The run can be
narrowed to specific workers (workers by name or nodeSelector by label, not both) and to specific devices.
apiVersion: storage.simplyblock.io/v1alpha2
kind: OperatorOps
metadata:
name: discover-rack-b
namespace: simplyblock
spec:
action: Discover
discover:
configName: rack-b-draft
nodeSelector:
storage.simplyblock.io/storage: "true"
deviceFilter:
pcieDenyList:
- "0000:00:1f.0"
driveSizeRange: 1T-4T
kubectl apply -f discover-rack-b.yaml
kubectl -n simplyblock get operatorops discover-rack-b -w
spec.discover narrows the run:
| Field | Value | Purpose |
|---|---|---|
configName |
— | The draft to write. Empty names it discovered-<run name>, so a second run never overwrites a draft that may already have been reviewed. |
workers |
Up to 128 node names | The workers to inspect. Empty inspects every schedulable worker. |
nodeSelector |
Labels | The workers to inspect, by label instead of by name. |
tolerations |
Up to 32 | What the probe pods tolerate. They reach the draft as well, because the taints a run probed through are the taints the cluster has to live with. |
enableControlPlaneNodes |
Off | Also considers nodes running the API server and etcd. |
deviceFilter |
— | Narrows the devices that reach the draft, see below. |
clusterRef |
— | Writes a growth draft for an existing cluster instead of a new one. |
workers and nodeSelector are mutually exclusive; a run that carries both is refused.
A probe pod is pinned to its worker with spec.nodeName, which bypasses the scheduler but not the taints. A run
against a dedicated storage plane that tolerates nothing therefore inspects nothing.
Device Filters
A run scans one class of device. The PCI filters select NVMe, which is the default, and the block filters require
enableLogicalBlockDevices. A filter for the class the run is not scanning is refused rather than ignored, so a
narrowed run is never silently un-narrowed.
| Field | Class | Purpose |
|---|---|---|
pcieAllowList, pcieDenyList |
NVMe | Select or exclude devices by PCI address. |
pcieModel |
NVMe | Select by model string. |
enableLogicalBlockDevices |
Block | Scans Linux block devices instead of NVMe. |
blockAllowList, blockDenyList |
Block | Select or exclude devices by path, for example, /dev/sdb. |
driveSizeRange |
Both | For example, 1T-4T. |
enablePartitionedDevices |
Block | Includes devices that already carry a partition table. |
enablePartitionedDevices works with Linux block devices only, not in an NVMe setup. The Helm value
discovery.enablePartitionedDevices therefore only takes effect together with discovery.enableLogicalBlockDevices.
All settings that depend on the device class are listed in
Settings by Device Class.
On a fleet that is uniform about which slot holds the boot device, one deny list keeps that device out of every group of every draft, which is the difference between correcting one document and correcting each of twenty groups.
The filters are inputs to the run only. The draft contains the explicit device list they produced, so re-running a filter against changed hardware cannot change what a reviewer already approved.
Review the Draft
A draft produced by discovery looks similar to the following. It lists every worker and device the run found and a proposed cluster template.
apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
name: discovered-initial-discovery
namespace: simplyblock
spec:
approved: false
environment: K3s
cluster:
name: discovered-initial-discovery-cluster
enableDriveFormat: true
maxSubsystemCount: 30
minHugePagesSize: 16G
vcpuCount: 16
stripe:
dataChunks: 1
parityChunks: 1
nodeSets:
- name: discovered
groups:
- name: group-1-nvme-4x3T
mgmtInterface: eth0
workers:
- worker-01
- worker-02
- worker-03
devices:
nvme:
- 0000:5e:00.0
- 0000:5f:00.0
- 0000:af:00.0
- 0000:b0:00.0
status.message of the draft lists validation findings, for example, DeviceNotFound, StripeBelowMinimumNodes,
or StripeBelowMinimumWorkers. The draft is edited in place until the findings are resolved:
kubectl -n simplyblock get cdc discovered-initial-discovery -o wide
kubectl -n simplyblock edit cdc discovered-initial-discovery
Environment
spec.environment names the Kubernetes distribution. Discovery detects it rather than asking for it, and the value
is spent once: it sets the storage node flags on the StorageCluster the document produces, after which nothing
reads it again.
| Value | What it sets on spec.storageNodes |
|---|---|
OpenShift |
The openshift block, enableCpuTopology, and enableKubeletConfiguration |
Talos |
Disables enableKubeletConfiguration, as Talos has no writable kubelet config |
Vanilla, Rancher, K3s |
enableKubeletConfiguration |
Cluster Template
spec.cluster is the template of the storage cluster to create. It is ignored when spec.clusterRef names an
existing cluster, because a growth document adds nodes to a cluster whose settings are already decided. The options
and which of them can change later are explained in Cluster Deployment Options.
This is the last chance to change these
Most of the fields below are immutable on the created StorageCluster. enableDriveFormat is also
destructive: it formats every device the document lists before a storage node takes it. Both are reasons the
approval is a deliberate step rather than a formality.
| Field | Value | Purpose |
|---|---|---|
name |
Required, at most 63 characters | Name of the StorageCluster. Discovery proposes <draft name>-cluster. |
maxSubsystemCount |
Required, 10 to 500 | NVMe subsystems the cluster serves. |
vcpuCount |
Required, at least 4 | vCPUs per storage node. Discovery proposes 15% of a socket, at least 5, see vCPU Count. |
minHugePagesSize |
For example, 16G |
Minimum huge page memory per storage node. |
stripe |
dataChunks and parityChunks |
The erasure coding scheme. Allowed: 1+0, 1+1, 2+1, 4+1, 1+2, 2+2, 4+2. |
fabricType |
For example, tcp |
The NVMe-oF transport. |
enableDriveFormat |
Off | Formats every listed device. Destructive. |
enableJournalDevice |
Off | Dedicates each worker's smallest NVMe device to the journal manager. |
socketsToUse, nodesPerSocket |
Up to 16 sockets, 1 to 8 nodes each | How many storage nodes run per worker. Empty socketsToUse means socket 0 only. |
nodeProvisioningBudget |
At least 1 | How many workers are provisioned in parallel, see Parallel Storage Node Addition. |
enableChecksumValidation, enableAtomicity4K |
Off | Data integrity. enableAtomicity4K requires checksum validation. |
enableFailureDomains |
Off | Spreads data across failure domains, see below. |
enableNodeAffinity |
Off | Keeps a volume's data on the node that owns it, see Configuring Node Affinity. |
kms |
— | External KMS for volume encryption keys, see Securing the Control Plane. |
backup |
— | The S3 store backups go to, see Backup and Recovery. |
ports |
— | Overrides the NVMe-oF, RPC, and node-agent ports. |
tolerations, containerResources, initContainerResources |
— | Placement and sizing of the storage node workload. |
vCPU Count
vcpuCount is the number of vCPUs every storage node of the cluster dedicates to the data plane. The value is the same
for all storage nodes of a cluster, and it is fixed when the cluster is created.
When the deployment document states no vcpuCount, discovery proposes one from the smallest worker it found:
- 15% of the logical CPUs (vCPUs) of the socket the storage node runs on, rounded down.
- At least 5 vCPUs.
- With several storage nodes per socket (
nodesPerSocket), the 15% are shared among them. - Never more than the socket provides beyond the two vCPUs reserved for the kubelet and the operating system, and never less than 4.
For example, a socket with 64 vCPUs gets 9, and a socket with 320 vCPUs gets 48. The proposal suits hyper-converged
workers, which share their CPUs with applications. Dedicated storage workers and I/O-intensive workloads benefit from a
higher count; see Storage Node CPU for the guidance. The
note in status.message of the draft states how the value was derived.
Operator images released before this fix
The operator images first released as 26.4.0-beta4, and earlier versions, propose all logical CPUs of the socket except two, for example, 318 on a socket with 320 vCPUs. On large workers, such a storage node cannot be scheduled and its addition to the cluster fails. With these images, the value must be checked and lowered in the draft before approval.
The proposal is changed in the draft before approval:
kubectl -n simplyblock patch cdc discovered-initial-discovery --type=merge -p '{"spec":{"cluster":{"vcpuCount":16}}}'
After approval, the vCPU count can no longer be changed. A cluster created with a wrong value, which holds no volumes
yet, is removed as described in Cleaning Up a Failed Deployment. The new discovery
run writes a new draft, whose vcpuCount is corrected before approval.
Node Groups and Devices
spec.nodeSets is how the workers are grouped. A node set becomes one storage cluster, and a group within it is the
set of workers that share one configuration. Workers whose hardware differs go in their own group rather than being
tuned individually. Racks and zones are groups of one node set, each with its own failureDomain.
A document with a single node set creates a standalone cluster. A document with several node sets creates a supercluster, with one placement group per node set, see Superclusters below.
| Level | Bound |
|---|---|
nodeSets |
1 to 64 |
groups per set |
1 to 64 |
workers per group |
1 to 200 |
Each group carries:
| Field | Purpose |
|---|---|
workers |
The Kubernetes node names. |
mgmtInterface, dataInterfaces |
The management and data network interfaces, up to 32 data interfaces. dataInterfaces only with spdkOnHostNetwork. |
devices |
Either nvme (PCI addresses) or block (for example, /dev/sdb), up to 128 each. Every group of a node set uses the same class. |
failureDomain |
The failure domain of all workers in the group, see below. |
spdkSystemMemory |
Huge page memory for the storage nodes of this group, for example, 8G. |
reservedSystemCPU |
CPUs reserved for the host, for example, 0,1 or 0-3. |
journalManager |
Journal copy count and per-device share. |
Workers or devices that should not become part of the cluster, for example, a boot disk, are removed from the draft before approving.
A storage cluster holds at most 32 storage nodes: the workers of a node set times the storage nodes per worker. A node
set above the limit is refused with MemberTooLarge.
Storage Array LUNs
On a storage array, the devices are the LUNs of each worker, listed
under devices.block. Block devices require enableJournalDevice, which dedicates the smallest LUN of each worker
to the journal. A discovery run with enableLogicalBlockDevices and forceJournalDevice drafts the document, or it
is written by hand.
Every worker has its own LUNs with their own WWNs, so the persistent device paths differ per worker. Each worker then gets a group of its own:
apiVersion: storage.simplyblock.io/v1alpha2
kind: ClusterDeploymentConfig
metadata:
name: array-cluster
namespace: simplyblock
spec:
approved: false
cluster:
name: array-cluster
enableJournalDevice: true
vcpuCount: 8
minHugePagesSize: 8G
stripe:
dataChunks: 4
parityChunks: 1
nodeSets:
- name: array
groups:
- name: worker-01
workers: [worker-01]
devices:
block:
- /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a01 # journal LUN, the smallest
- /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a02 # data LUN
- name: worker-02
workers: [worker-02]
devices:
block:
- /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a11
- /dev/disk/by-id/wwn-0x600a098038304331395d4f4a2d2f4a12
# ... one group per worker, at least six workers for 4+1
The LUN layout, the erasure coding scheme, and the multipathing requirements are described in Storage Arrays.
Failure Domains
With spec.cluster.enableFailureDomains: true, every group must carry a failureDomain label, for example, a rack
name. The label is copied to each generated StorageNode and cannot be changed afterward.
spec:
cluster:
name: production
maxSubsystemCount: 50
vcpuCount: 8
stripe:
dataChunks: 2
parityChunks: 1
enableFailureDomains: true
nodeSets:
- name: racks
groups:
- name: rack-a
failureDomain: rack-a
workers: [worker-1, worker-2]
devices:
nvme: ["0000:01:00.0", "0000:02:00.0"]
- name: rack-b
failureDomain: rack-b
workers: [worker-3, worker-4]
devices:
nvme: ["0000:01:00.0", "0000:02:00.0"]
- name: rack-c
failureDomain: rack-c
workers: [worker-5, worker-6]
devices:
nvme: ["0000:01:00.0", "0000:02:00.0"]
Failure-domain mode needs at least as many failure domains as parity chunks plus two, with the same number of
workers in each. A document that does not meet this is refused with FailureDomainLayout.
For details, see Failure Domains.
Superclusters
A fleet that does not fit into one storage cluster is deployed as a supercluster. Discovery drafts one on its own
when the workers exceed 32 storage nodes, span several regions, or differ in network interfaces or device class. The
draft then holds several node sets, and its status.message names the placement groups the approval creates.
Each node set becomes a placement group: a complete storage cluster
named <cluster>-<nodeset>-<suffix>. The supercluster takes the name in spec.cluster.name, and storage pools and
StorageClasses name it like any cluster. Every new volume is placed into one placement group. A node set can override
the sizing and the erasure coding scheme of the template for its own placement group.
The topology, the sizing, and the validation of a supercluster are described in Superclusters and Managing Superclusters.
Copying, Splitting, and Hand-Authoring
A ClusterDeploymentConfig is an ordinary Kubernetes resource, and the discovered draft is a starting point rather
than the only thing that may be approved. A reviewer can copy it under another name, split one draft into several
documents that are approved separately, or write one from scratch and never run discovery at all.
Splitting is the usual reason. A draft covering three racks can become three documents, each holding one rack's
groups, so each rack is reviewed and brought up on its own schedule. The first document creates the cluster and the
other two name it in spec.clusterRef.
A copy is a plain kubectl get -o yaml, edited and applied under a new name. Clear metadata.resourceVersion,
metadata.uid, and status from it first, as with any copied resource.
A document discovery wrote cannot be wrong about its devices, because the list came from the inspection. A hand-written one can, and the validation above is the only check a device gets before the deployment runs.
When Does the Cluster Become Active?
The erasure coding scheme determines the minimum number of storage nodes: enough to place a stripe across, which is the data chunks plus the parity chunks, plus one spare per tolerated failure to rebuild onto.
| Scheme | Minimum storage nodes |
|---|---|
| 1+0 | 1 |
| 1+1 | 3 |
| 2+1 | 4 |
| 1+2 | 5 |
| 4+1 | 6 |
| 2+2 | 6 |
| 4+2 | 8 |
The approval is refused if the document provides fewer nodes, with StripeBelowMinimumNodes, or if they sit on too
few workers, with StripeBelowMinimumWorkers. The second is counted separately because a fleet that reaches the
number by running several nodes per socket on two workers has not bought the independent spare the count asks for:
every node of a worker fails with the worker. An unstated scheme counts as 1+1.
The operator activates the cluster as soon as all storage nodes of the document are Online. In a supercluster,
the minimum applies to each node set, and each placement group is activated once its own storage nodes are Online.
Approve the Deployment
Once the draft is correct, it is approved by setting spec.approved: true. The admission webhook validates the
approving edit against the live cluster, for example, that the devices exist. After approval, the document is
immutable and the approval cannot be withdrawn.
kubectl -n simplyblock patch cdc discovered-initial-discovery \
--type=merge -p '{"spec":{"approved":true}}'
kubectl -n simplyblock get cdc discovered-initial-discovery -w
The document moves from Draft to Expanding and finally to Expanded (or Failed). While expanding, it passes the
steps Validating, CreatingCluster, AwaitingCluster, CreatingNodes, and Activating.
Warning
Simplyblock exclusively owns the resources it has been allocated. It must be ensured they are sized correctly alongside other workloads.
Additionally, simplyblock manages huge page allocation automatically. Total RAM required depends on vCPU count, the number of active logical volumes, and utilized virtual storage per node.
More information can be found in Minimum Hardware Requirements.
What Gets Created
The expansion creates the following resources:
StorageCluster: Named afterspec.cluster.name, in the namespace of the document. For a supercluster, one furtherStorageClusteris created per node set as its placement group.StoragePool<cluster>-default: The default pool. It is created once and never recreated.- StorageClass
simplyblock-<namespace>-<cluster>: The class of the default pool, with the provisionercsi.simplyblock.io,WaitForFirstConsumerbinding, the reclaim policyDelete, and volume expansion enabled. It is not marked as the Kubernetes default class. StorageNoderesources: One per worker, socket, and node per socket. Each is provisioned by the storage node DaemonSet, and its devices appear asStorageDeviceresources.StorageClusterOps<cluster>-activate: Raised once all storage nodes areOnline.
With a cluster named production in the namespace simplyblock, the default pool is production-default and the
StorageClass is simplyblock-simplyblock-production.
The ClusterDeploymentConfig is no longer needed after it reaches Expanded and can be deleted.
Verify the Cluster
kubectl -n simplyblock get storagecluster,storagenode,storagedevice,storagepool
The same resources can be listed with their short names, kubectl -n simplyblock get stc,sn,sd,sp.
NAME PHASE ROLE STEP STATUS EC UUID AGE
production Online Standalone active 2x1 bfa260ce-06a7-4bcb-a843-813d0be633af 12m
The StorageCluster phase is Online once the cluster is active. Degraded and Unavailable indicate problems,
and status.message explains them. The storage nodes report Online in their phase column.
The operator also writes the cluster credentials for the CSI driver into the Secret simplyblock-csi-secret-v2 in the
operator's namespace. The Secret is maintained by the operator and does not need to be managed manually.
Cleaning Up a Failed Deployment
A deployment that never reached Online (for example, because a storage node could not be added) is cleaned up in
one of two ways:
- Resource cleanup removes only the storage cluster and its documents. The operator, the control plane, FoundationDB, the CSI driver, and the certificates stay in place, and discovery and deployment are run again. It is the faster way and is tried first.
- Full cleanup removes the whole installation, including the operator, and is followed by a new installation. It is used when the resource cleanup does not complete, or when the installation itself is broken.
Both apply only to a cluster that holds no volumes yet.
The reason of the failure is found and fixed first, otherwise the next attempt fails the same way. Typical signs are a
StorageCluster that stays in the phase Provisioning, a node_add task with the status suspended, and
NodeAddFailing events on the storage nodes:
kubectl -n simplyblock get storagecluster -o yaml | grep -A12 "^ status:"
kubectl -n simplyblock get events --field-selector reason=NodeAddFailing
kubectl -n simplyblock get events --field-selector reason=FailedScheduling
Resource Cleanup for Rerunning Discovery and Deployment
The resources are removed in order, each one after the previous is gone. Deleting the StorageCluster also deletes
the cluster in the control plane:
NS=simplyblock
kubectl -n $NS delete storagenodes --all --wait=true --timeout=10m
kubectl -n $NS delete storagenodesets --all --wait=true --timeout=10m
kubectl -n $NS delete storagecluster --all --wait=true --timeout=10m
kubectl -n $NS delete clusterdeploymentconfigs --all --wait=true
kubectl -n $NS delete operatorops --all --wait=true
kubectl -n $NS get storagecluster,storagenodes,storagenodesets,clusterdeploymentconfigs,operatorops
On OpenShift, the machine configuration of the failed storage nodes is removed as well (see Machine Configuration of a Failed Deployment).
With no StorageCluster, ClusterDeploymentConfig, or OperatorOps left, the operator starts the
initial discovery again and writes a new draft. The draft is corrected (for example,
its vCPU count) and approved as described above.
A resource that stays in Terminating after the timeout has a finalizer the operator cannot clear, typically because
the control plane refused to delete the cluster or a half-added storage node. The operator log names the reason. If it
cannot be resolved, the full cleanup is used instead.
Full Cleanup for Redeployment
The whole installation is removed with the cleanup script of the Simplyblock Operator (see Removing a Previous Installation), which deletes the custom resources, their finalizers, the operator, node labels, and persistent volumes:
curl -sO https://raw.githubusercontent.com/simplyblock/simplyblock-operator/main/helm-charts/scripts/cleanup-simplyblock.sh
chmod +x cleanup-simplyblock.sh
export KUBECONFIG=/path/to/kubeconfig
./cleanup-simplyblock.sh simplyblock
The script does not remove machine configuration on OpenShift, which is removed as described in Machine Configuration of a Failed Deployment.
The operator is then installed again (see Installing the Operator). The initial discovery runs as on a new installation and writes a new draft, which is corrected and approved.
Create a Storage Pool
Volumes can be provisioned from the default pool right away. Additional pools divide the cluster into tenancy units
with their own capacity and QoS limits. A StoragePool references its cluster with clusterRef.
apiVersion: storage.simplyblock.io/v1alpha2
kind: StoragePool
metadata:
name: tenant-a
namespace: simplyblock
spec:
clusterRef: production
limits:
capacity: 10T
maxVolumeSize: 2T
iops: 200000
throughput:
readWrite: 4096
volumeDefaults:
iops: 20000
throughput:
readWrite: 512
filesystem: xfs
kubectl apply -f storage-pool.yaml
kubectl -n simplyblock get storagepool tenant-a
spec.limits (capacity, maximum volume size, IOPS, and throughput in MB/s) can be changed later. spec.volumeDefaults
is immutable once set. spec.allowedNodes restricts the pool to specific Kubernetes nodes, see
Host Authentication and Encryption.
Deleting a Pool
A pool carries the finalizer storage.simplyblock.io/storagepool-finalizer and refuses to finish deleting while
anything Kubernetes knows about still refers to it.
| Deleting a pool with | Result |
|---|---|
An authored StorageClass assigned to it |
Held. StorageClassStillAssigned, requeued, nothing deleted |
| Only the operator's own default class | The class is deleted, then the backend pool, then the finalizer clears |
A PersistentVolume in it |
Held. VolumesStillBound, requeued, nothing deleted |
| Neither | The backend pool is deleted and the finalizer clears |
An authored class holds the deletion because the operator neither wrote it nor knows why it exists, and deleting somebody's provisioning contract to let a pool go is not a trade it makes. Deleting the class releases the pool. The operator's own default class does not hold, because removing it is cleanup rather than a decision.
A StorageCluster owns its pools, so deleting one cascades, and is held behind any pool that is itself held.
Author a StorageClass for the Pool
The operator creates a StorageClass only for the default pool. For every other pool, the StorageClass is written by
the administrator and linked to the pool with three labels: storage.simplyblock.io/namespace,
storage.simplyblock.io/cluster, and storage.simplyblock.io/pool. The parameters cluster_id (the cluster's
status.uuid) and pool_name direct the CSI driver to the pool.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: tenant-a-fast
labels:
storage.simplyblock.io/namespace: simplyblock
storage.simplyblock.io/cluster: production
storage.simplyblock.io/pool: tenant-a
provisioner: csi.simplyblock.io
parameters:
cluster_id: "<StorageCluster status.uuid>"
pool_name: tenant-a
max_iops: "20000"
csi.storage.k8s.io/fstype: xfs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
The pool lists its assigned classes in status.storageClassNames. For the full parameter mapping, see
Storage Class.
Provision the First Volume
Now, everything is in place to create the first volume. The operator has deployed the simplyblock CSI driver into the Kubernetes cluster. Hence, creating a volume is as simple as creating a PersistentVolumeClaim with the correct StorageClass set.
Create the PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: simplyblock-test-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: simplyblock-simplyblock-production
kubectl apply -f test-pvc.yaml
kubectl get pvc simplyblock-test-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
simplyblock-test-pvc Pending simplyblock-simplyblock-production 5s
Since provisioning is asynchronous, the PVC status is initially Pending. The StorageClass uses
WaitForFirstConsumer, which means the volume is not provisioned until a pod actually needs it. The scheduler picks
the right node first, then the volume is created close to where it will be used.
Mount the Volume into a Test Pod
To mount the volume, it can be used like any other Kubernetes persistent volume claim by referencing it in the pod's volumes specification.
apiVersion: v1
kind: Pod
metadata:
name: simplyblock-test-pod
spec:
containers:
- name: test
image: busybox
command: ["/bin/sh", "-c", "echo 'volume provisioned successfully' > /data/test.txt && sleep 3600"]
volumeMounts:
- mountPath: /data
name: storage
volumes:
- name: storage
persistentVolumeClaim:
claimName: simplyblock-test-pvc
kubectl apply -f test-pod.yaml
When the pod reaches Running status, the PVC changes to bound.
kubectl get pvc simplyblock-test-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
simplyblock-test-pvc Bound pvc-3f2a1c9e-84b1-4d2e-9f3a-1234abcd5678 10Gi RWO simplyblock-simplyblock-production 30s
It is now possible to access the data written to the volume when the pod started up.
kubectl exec simplyblock-test-pod -- cat /data/test.txt
volume provisioned successfully
The cluster is now fully operational and the test resources should be cleaned up with:
kubectl delete pod simplyblock-test-pod
kubectl delete pvc simplyblock-test-pvc
Next Steps
The cluster serves volumes now. Two things it does not yet have are more capacity and a second cluster, and both are their own deployment document.
More nodes for this cluster. A ClusterDeploymentConfig that names the cluster in spec.clusterRef, and
carries no spec.cluster block, adds nodes to it. A discovery run with spec.discover.clusterRef writes such a
growth draft from whatever workers are still unclaimed. See
Expanding a Storage Cluster.
A second storage cluster. Several storage clusters can share a namespace, and each one is pointed at a different set of workers. Clusters that are meant to act as one are deployed as a supercluster instead, see Managing Superclusters.
From here, Operations covers running the cluster, and Usage covers provisioning volumes from it.