Create a Storage Cluster
With the Simplyblock Operator being installed, it's time to bring up a storage cluster.
This includes creating the cluster resource, adding storage nodes, creating a storage pool, and provisioning the first simplyblock logical volume.
Before going on, here is a high-level overview of the following deployment process:
StorageCluster ──► unready
│
▼ (enroll ≥ 3 workers in a StorageNodeSet)
StorageNodeSet ──► operator creates StorageNode CRs and provisions each worker
StorageNode(s) ──► active (once ≥ 3 nodes are online)
│
▼ (create a pool)
Pool ──► StorageClass created automatically
│
▼ (create a PVC)
PersistentVolume ──► Bound
Info
Not all Kubernetes workers have to become part of the simplyblock storage cluster. It is possible and common to only use a subset of all Kubernetes worker nodes for storage.
It is also possible to use a separate Kubernetes worker node pool dedicated to storage. In this case, it is important to remember to taint the nodes accordingly to prevent other workloads from being scheduled on them.
Prerequisites
OpenShift
If deploying onto an OpenShift cluster, there are additional environment-specific steps in the OpenShift Installation guide before continuing here.
Talos
If deploying onto a Talos cluster, there are additional environment-specific steps in the Talos Installation guide before continuing here.
Networking
Multiple ports must be open on storage node hosts.
It is required to use one or more separate VLANs for simplyblock. Ports within the same VLAN do not require extra firewall rules, but ports between the control plane and storage networks typically do.
| Service | Direction | Hosts | Network | Port(s) | Protocol(s) |
|---|---|---|---|---|---|
| ICMP | egress | control | Control | - | ICMP |
| storage-node-api | egress | storage | Control | 5000 | TCP |
| NVMf (client-target) | egress | client | Storage | 4420-4499 | TCP, UDP (1) |
| FoundationDB | ingress, egress | control | Control | 4500 | TCP |
| control-plane-API | ingress | control | Control | 80 | TCP |
| storage-node-RPC | ingress, egress | storage, control | Control | 8080-9044 | TCP |
| Monitoring Stack (2) | ingress, egress | control | Control | 12202, 13301, 13302, 9200, 9090 | TCP |
1 The NVMf port range carries both transports. NVMe-oF/TCP uses it over TCP, NVMe-oF/RDMA
over UDP. Only the protocol of the transport(s) in use has to be opened.
2 The monitoring stack is optional. Its ports only have to be opened where it is deployed.
Create the Storage Cluster
The first step is to create a StorageCluster resource. This registers the cluster with the operator and prepares the
control plane. This step does not yet acquire storage devices.
apiVersion: storage.simplyblock.io/v1alpha1
kind: StorageCluster
metadata:
name: simplyblock-cluster
namespace: simplyblock
spec:
fabricType: tcp
maxSubsystemCount: 75
vcpuCount: 16
stripe:
dataChunks: 2
parityChunks: 1
kubectl apply -f storage-cluster.yaml
Next, the cluster status can be checked:
kubectl get storagecluster -n simplyblock
The output should look similar to this:
NAME STATUS UUID CONFIGURED AGE
simplyblock-cluster unready 81932010-8c06-4acd-b14a-51f5c3fca425 true 1m
The cluster is set up, but not yet ready to use. Hence, unready is expected at this point. While the cluster has
been registered, it has no storage nodes yet. Those are added in the next step.
Note
There are additional configuration properties when creating a storage cluster. The documentation, such as NVMe-oF transport security, backup configuration, capacity thresholds, and more, are available at Cluster Deployment Options.
External KMS
If volumes in this cluster should offload their encryption keys to an external KMS, set
spec.hashicorpVaultSettings.baseURL on the StorageCluster now. The setting can also be added later, but
configuring it upfront means encrypted volumes use the external KMS from day one. See
Securing the Control Plane: External KMS.
Add Storage Nodes
To enroll Kubernetes workers as simplyblock storage nodes, create a StorageNodeSet resource. It declares which
workers to use and how to configure them. The operator creates one StorageNode CR per worker (and per NUMA socket
if multiple sockets are configured), then provisions each one sequentially.
apiVersion: storage.simplyblock.io/v1alpha1
kind: StorageNodeSet
metadata:
name: simplyblock-node
namespace: simplyblock
spec:
clusterName: simplyblock-cluster
workerNodes:
- worker-1.example.com
- worker-2.example.com
- worker-3.example.com
kubectl apply -f storage-nodeset.yaml
As part of the provisioning process, the operator bootstraps each listed worker, installs the SPDK service, and
registers it with the previously created storage cluster. Workers are added one at a time by default
(maxParallelNodeAdds: 1) to protect FoundationDB from simultaneous reboots.
The process takes a little while as SPDK pods are created. Track progress with:
kubectl get storagenodeset simplyblock-node -n simplyblock
kubectl get storagenodes -n simplyblock
When does the Cluster become Active?
By default, simplyblock clusters use the Erasure Coding schema
of 1+1 which requires at least three storage nodes to join the cluster.
The operator automatically activates the cluster when at least three storage nodes are online. For other erasure coding schemes, the required number differs (see the erasure coding documentation for details).
kubectl get storagecluster -n simplyblock
NAME STATUS UUID CONFIGURED AGE
simplyblock-cluster active bfa260ce-06a7-4bcb-a843-813d0be633af true 10m
When the status becomes active, the operator automatically creates a simplyblock-csi-secret-v2 secret in the
simplyblock namespace, containing the cluster credentials for the CSI driver.
There is no necessity to manage this secret manually. The operator keeps it up to date and removes the cluster entry when the cluster is deleted.
For a full list of configuration options see Simplyblock Operator: StorageNodeSet.
Warning
Simplyblock exclusively owns the resources it has been allocated. It must be ensured they are sized correctly alongside other workloads.
Additionally, simplyblock manages huge page allocation automatically. Total RAM required depends on vCPU count, the number of active logical volumes, and utilized virtual storage per node.
More information can be found in Minimum Hardware Requirements.
Create a Storage Pool
A storage pool is a grouping of logical volumes and capacity limits within the cluster. An initial StoragePool resource must
be created to define a storage pool before being able to provision volumes.
apiVersion: storage.simplyblock.io/v1alpha1
kind: StoragePool
metadata:
name: production-pool
namespace: simplyblock
spec:
clusterName: production
capacityLimit: "10T"
kubectl apply -f storage-pool.yaml
The status of the storage pool can be checked with:
kubectl get storagepools -n simplyblock
Once the pool is active, the operator automatically creates a StorageClass named
simplyblock-<namespace>-<clusterName>-<poolName>. In this example, the StorageClass is called
simplyblock-simplyblock-cluster-production-pool.
cluster_id and pool_name are set from the storage pool and cannot be overridden. The remaining StorageClass
parameters are copied from spec.storageClassParameters. See
Storage Class: StorageClass Created by a Storage Pool
for the full parameter mapping.
A StorageClass's parameters cannot be changed after creation, so spec.storageClassParameters is immutable
once the storage pool is created. A new storage pool is required to provision volumes with different defaults.
Pool limits are read once
capacityLimit, logicalVolumeMaxSize, and qos are sent to the control plane when the pool is created and are
not reconciled afterward. Patching one of them on an existing StoragePool is accepted by the API server and has
no effect on the pool, so a different capacity limit or a different set of QoS limits requires a new storage pool.
allowedNodes is the exception and is reconciled, see
Host Authentication and Encryption.
The StorageClass is automatically removed when the storage pool is deleted. Full details and customization options are available at Simplyblock Operator: Storage Pool.
kubectl get storageclass simplyblock-simplyblock-production-my-pool
Provision the First Volume
Now, everything is in place to create the first volume. The operator has automatically deployed the Simplyblock CSI Driver into the Kubernetes cluster. Hence, creating a volume is as simple as creating PersistentVolumeClaim with the correct StorageClass set.
Create the PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: simplyblock-test-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: simplyblock-simplyblock-production-my-pool
kubectl apply -f test-pvc.yaml
kubectl get pvc simplyblock-test-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
simplyblock-test-pvc Pending simplyblock-simplyblock-production-my-pool 5s
Since provisioning is asynchronous, the PVC status will initially be Pending. The StorageClass uses
WaitForFirstConsumer by default, which means the volume is not provisioned until a pod actually needs it. The
scheduler picks the right node first, then the volume is created close to where it will be used.
Mount the Volume into a Test Pod
To mount the volume, it can be used like any other Kubernetes persistent volume claim by referencing it in the pod's volumes specification.
apiVersion: v1
kind: Pod
metadata:
name: simplyblock-test-pod
spec:
containers:
- name: test
image: busybox
command: ["/bin/sh", "-c", "echo 'volume provisioned successfully' > /data/test.txt && sleep 3600"]
volumeMounts:
- mountPath: /data
name: storage
volumes:
- name: storage
persistentVolumeClaim:
claimName: simplyblock-test-pvc
kubectl apply -f test-pod.yaml
When the pod reaches Running status, the PVC changes to bound.
kubectl get pvc simplyblock-test-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
simplyblock-test-pvc Bound pvc-3f2a1c9e-84b1-4d2e-9f3a-1234abcd5678 10Gi RWO simplyblock-simplyblock-production-my-pool 30s
It is now possible to access the data written to the volume when the pod started up.
kubectl exec simplyblock-test-pod -- cat /data/test.txt
volume provisioned successfully
The cluster is now fully operational and the test resources should be cleaned up with:
kubectl delete pod simplyblock-test-pod
kubectl delete pvc simplyblock-test-pvc
Multi-Cluster Storage Node Support
A single Kubernetes cluster can host storage nodes connected to multiple simplyblock clusters.
Create a separate StorageNodeSet for each simplyblock cluster. Multiple storage clusters can share Kubernetes
worker nodes, but it is recommended to point each storage cluster to a different set of workers.
apiVersion: storage.simplyblock.io/v1alpha1
kind: StorageNodeSet
metadata:
name: cluster-a-nodes
namespace: simplyblock
spec:
clusterName: cluster-a
workerNodes:
- worker-a-1.example.com
- worker-a-2.example.com
---
apiVersion: storage.simplyblock.io/v1alpha1
kind: StorageNodeSet
metadata:
name: cluster-b-nodes
namespace: simplyblock
spec:
clusterName: cluster-b
workerNodes:
- worker-b-1.example.com
- worker-b-2.example.com