Hyper-Converged Setup
Control Plane Installation
The first step when installing simplyblock, is to install the control plane. The control plane manages one or more storage clusters. If an existing control plane is available and the new cluster should be added to it, this section can be skipped. Jump right to the Storage Plane Installation.
Firewall Configuration (CP)
Simplyblock requires a number of TCP and UDP ports to be opened from certain networks. Additionally, it requires IPv6 to be disabled on management nodes.
Following is a list of all ports (TCP and UDP) required for operation as a management node. Attention is required, as this list is for management nodes only. Storage nodes have a different port configuration. See the Firewall Configuration section for the storage plane.
| Service | Direction | Source / Target Network | Port | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| Cluster API | ingress | storage, control, admin | 80 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Graylog | ingress | storage, control | 12201 | TCP / UDP |
| Graylog | ingress | storage, control | 12202 | TCP |
| Graylog | ingress | storage, control | 13201 | TCP |
| Graylog | ingress | storage, control | 13202 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| FoundationDB | ingress | storage, control | 4500 | TCP |
| Prometheus | ingress | storage, control | 9100 | TCP |
| Cluster Control | egress | storage, control | 8080-8890 | TCP |
| spdk-http-proxy | egress | storage, control | 5000 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
With the previously defined subnets, the following snippet disables IPv6 and configures the iptables automatically.
Danger
The example assumes that you have an external firewall between the admin network and the public internet!
If this is not the case, ensure the correct source access for ports 22 and 80.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
# Clean up
sudo iptables -F SIMPLYBLOCK
sudo iptables -D DOCKER-FORWARD -j SIMPLYBLOCK
sudo iptables -X SIMPLYBLOCK
# Setup
sudo iptables -N SIMPLYBLOCK
sudo iptables -I DOCKER-FORWARD 1 -j SIMPLYBLOCK
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A SIMPLYBLOCK -m state --state ESTABLISHED,RELATED -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 80 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2375 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2377 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 4500 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 4789 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 9100 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 12201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 12201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 12202 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 13201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 13202 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -s 0.0.0.0/0 -j DROP
Management Node Installation
Now that the network is configured, the management node software can be installed.
Simplyblock provides a command line interface called sbcli-pre. It's built in Python and required Python 3 and Pip (the
Python package manager) installed on the machine. This can be achieved with yum.
sudo yum -y install python3-pip
Afterward, the sbcli-pre command line interface can be installed. Upgrading the CLI later on, uses the same command.
sudo pip install sbcli-pre --upgrade
Recommendation
Simplyblock recommends to only upgrade sbcli-pre if a system upgrade is executed to prevent potential
incompatibilities between the running simplyblock cluster and the version of sbcli-pre.
At this point, a quick check with the simplyblock provided system check can reveal potential issues quickly.
curl -L https://sblk.xyz/prerequisites | bash
If the check succeeds, it's time to set up the primary management node:
sbcli-pre cluster create --ifname=<IF_NAME> --ha-type=ha
The output should look something like this:
[root@vm11 ~]# sbcli-pre cluster create --ifname=eth0 --ha-type=ha
2025-02-26 12:37:06,097: INFO: Installing dependencies...
2025-02-26 12:37:13,338: INFO: Installing dependencies > Done
2025-02-26 12:37:13,358: INFO: Node IP: 192.168.10.1
2025-02-26 12:37:13,510: INFO: Configuring docker swarm...
2025-02-26 12:37:14,199: INFO: Configuring docker swarm > Done
2025-02-26 12:37:14,200: INFO: Adding new cluster object
File moved to /usr/local/lib/python3.9/site-packages/simplyblock_core/scripts/alerting/alert_resources.yaml successfully.
2025-02-26 12:37:14,269: INFO: Deploying swarm stack ...
2025-02-26 12:38:52,601: INFO: Deploying swarm stack > Done
2025-02-26 12:38:52,604: INFO: deploying swarm stack succeeded
2025-02-26 12:38:52,605: INFO: Configuring DB...
2025-02-26 12:39:06,003: INFO: Configuring DB > Done
2025-02-26 12:39:06,106: INFO: Settings updated for existing indices.
2025-02-26 12:39:06,147: INFO: Template created for future indices.
2025-02-26 12:39:06,505: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "Cluster", "message": "Cluster created 7bef076c-82b7-46a5-9f30-8c938b30e655", "caused_by": "cli"}
2025-02-26 12:39:06,529: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "MgmtNode", "message": "Management node added vm11", "caused_by": "cli"}
2025-02-26 12:39:06,533: INFO: Done
2025-02-26 12:39:06,535: INFO: New Cluster has been created
2025-02-26 12:39:06,535: INFO: 7bef076c-82b7-46a5-9f30-8c938b30e655
7bef076c-82b7-46a5-9f30-8c938b30e655
If the deployment was successful, the last line returns the cluster id. This should be noted down. It's required in further steps of the installation.
Additionally to the cluster id, the cluster secret is required in many further steps. The following command can be used to retrieve it.
sbcli-pre cluster get-secret <CLUSTER_ID>
[root@vm11 ~]# sbcli-pre cluster get-secret 7bef076c-82b7-46a5-9f30-8c938b30e655
e8SQ1ElMm8Y9XIwyn8O0
Secondary Management Nodes
A production cluster, requires at least three management nodes in the control plane. Hence, additional management nodes need to be added.
On the secondary nodes, the network requires the same configuration as on the primary. Executing the commands under Firewall Configuration (CP) will get the node prepared.
Afterward, Python, Pip, and sbcli-pre need to be installed.
sudo yum -y install python3-pip
pip install sbcli-pre --upgrade
Finally, we deploy the management node software and join the control plane cluster.
sbcli-pre mgmt add <CP_PRIMARY_IP> <CLUSTER_ID> <CLUSTER_SECRET> <IF_NAME>
Running against the primary management node in the control plane should create an output similar to the following example:
[demo@demo ~]# sbcli-pre mgmt add 192.168.10.1 7bef076c-82b7-46a5-9f30-8c938b30e655 e8SQ1ElMm8Y9XIwyn8O0 eth0
2025-02-26 12:40:17,815: INFO: Cluster found, NQN:nqn.2023-02.io.simplyblock:7bef076c-82b7-46a5-9f30-8c938b30e655
2025-02-26 12:40:17,816: INFO: Installing dependencies...
2025-02-26 12:40:25,606: INFO: Installing dependencies > Done
2025-02-26 12:40:25,626: INFO: Node IP: 192.168.10.2
2025-02-26 12:40:26,802: INFO: Joining docker swarm...
2025-02-26 12:40:27,719: INFO: Joining docker swarm > Done
2025-02-26 12:40:32,726: INFO: Adding management node object
2025-02-26 12:40:32,745: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "MgmtNode", "message": "Management node added vm12", "caused_by": "cli"}
2025-02-26 12:40:32,752: INFO: Done
2025-02-26 12:40:32,755: INFO: Node joined the cluster
cdde125a-0bf3-4841-a6ef-a0b2f41b8245
From here, additional management nodes can be added to the control plane cluster. If the control plane cluster is ready, the storage plane can be installed.
Storage Plane Installation
Caching nodes, like storage nodes, require huge page memory to hold the internal state. Huge pages should be 2MiB in size and a minimum of 4096 huge pages should be allocated at boot time of the operating system.
demo@worker-1 ~> sudo sysctl -w vm.nr_hugepages=4096
Info
To see how huge pages can be pre-reserved at boot time, see the node sizing documentation section on
Huge Pages.
demo@worker-1 ~> sudo systemctl restart kubelet
demo@worker-1 ~> kubectl describe node worker-1.kubernetes-cluster.local | \
grep hugepages-2Mi
demo@demo ~> kubectl describe node worker-1.kubernetes-cluster.local | \
grep hugepages-2Mi
hugepages-2Mi: 9440Mi
hugepages-2Mi: 9440Mi
hugepages-2Mi 0 (0%) 0 (0%)
demo@worker-1 ~> sudo yum install -y nvme-cli
demo@worker-1 ~> sudo modprobe nvme-tcp
demo@worker-1 ~> sudo modprobe nbd
Firewall Configuration (SP)
| Service | Direction | Source / Target Network | Port(s) | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| bdts | ingress | storage | 4420 | TCP |
| Cluster Control | ingress | control | 5000 | TCP |
| spdk-http-proxy | ingress | storage, control | 8080 | TCP |
| lvol-proxy | ingress | storage, control | 9090-9900 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| FoundationDB | egress | storage | 4500 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
| Graylog | egress | control | 12202 | TCP |
Storage Node Installation
Simplyblock is built upon the NVMe over Fabrics standard and uses NVMe over TCP (NVMe/TCP) by default.
While the driver is part of the Linux kernel with kernel versions 5.x and later, it is not enabled by default. Hence, when using simplyblock, the driver needs to be loaded.
modprobe nvme-tcp
When loading the NVMe/TCP driver, the NVMe over Fabrics driver automatically get loaded to, as the former depends on its provided foundations.
It is possible to check for successful loading of both drivers with the following command:
lsmod | grep 'nvme_'
The response should list the drivers as nvme_tcp and nvme_fabrics as seen in the following example:
[demo@demo ~]# lsmod | grep 'nvme_'
nvme_tcp 57344 0
nvme_keyring 16384 1 nvme_tcp
nvme_fabrics 45056 1 nvme_tcp
nvme_core 237568 3 nvme_tcp,nvme,nvme_fabrics
nvme_auth 28672 1 nvme_core
t10_pi 20480 2 sd_mod,nvme_core
To make the driver loading persistent and survive system reboots, it has to be configured to be loaded at system startup time. This can be achieved by either adding it to /etc/modules (Debian / Ubuntu) or creating a config file under /etc/modules-load.d/ (Red Hat / Alma / Rocky).
echo "nvme-tcp" | sudo tee -a /etc/modules-load.d/nvme-tcp.conf
echo "nvme-tcp" | sudo tee -a /etc/modules
After rebooting the system, the driver should be loaded automatically. It can be checked again via the above provided
lsmod command.
To install the simplyblock in Kubernetes, a helm chart is provided. While it can be installed manually, the helm chart is strongly recommended. The installation requires a few values to be available.
First we need the unique cluster id. Note down the cluster uuid of the cluster to access.
sudo sbcli-pre cluster list
An example of the output is below.
[demo@demo ~]# sbcli-pre cluster list
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
| UUID | NQN | ha_type | tls | mgmt nodes | storage nodes | Mod | Status |
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
| 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a | nqn.2023-02.io.simplyblock:4502977c-ae2d-4046-a8c5-ccc7fa78eb9a | ha | False | 1 | 4 | 1x1 | active |
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
In addition, we need the cluster secret. Note down the cluster secret.
sbcli-pre cluster get-secret <CLUSTER_UUID>
Retrieving the cluster secret will look somewhat like that.
[demo@demo ~]# sbcli-pre cluster get-secret 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a
oal4PVNbZ80uhLMah2Bs
Additionally, a storage pool is required. If a pool already exists, it can be reused. Otherwise, creating a storage pool can be created as following:
sbcli-pre pool add <POOL_NAME> <CLUSTER_UUID>
The last line of a successful storage pool creation returns the new pool id.
[demo@demo ~]# sbcli-pre pool add test 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a
2025-03-05 06:36:06,093: INFO: Adding pool
2025-03-05 06:36:06,098: INFO: {"cluster_id": "4502977c-ae2d-4046-a8c5-ccc7fa78eb9a", "event": "OBJ_CREATED", "object_name": "Pool", "message": "Pool created test", "caused_by": "cli"}
2025-03-05 06:36:06,100: INFO: Done
ad35b7bb-7703-4d38-884f-d8e56ffdafc6 # <- Pool Id
The last item necessary before deploying simplyblock is the control plane address. This is any of the API addresses of a
management node. Meaning, if the primary management node has the IP of 192.168.10.1, the control plane address is
http://192.168.0.1. It is, however, recommended to front all management nodes, with a load balancing proxy, such as
HAproxy. In the latter case, the load balancer URL would be the address of the control plane.
Anyhow, deploying simplyblock using the provided helm chart comes down to providing the four necessary values, adding the helm chart repository, and installing the driver. In addition to the storage nodes, this will also install the Simplyblock CSI driver for seamless integration with the Kubernetes CSI persistent storage subsystem.
CLUSTER_UUID="<UUID>"
CLUSTER_SECRET="<SECRET>"
CNTR_ADDR="<CONTROL-PLANE-ADDR>"
POOL_NAME="<POOL-NAME>"
helm repo add simplyblock-csi https://install.simplyblock.io/helm
helm repo update
helm install -n simplyblock-csi \
--create-namespace simplyblock-csi \
simplyblock-csi/spdk-csi \
--set csiConfig.simplybk.uuid=<CLUSTER_UUID> \
--set csiConfig.simplybk.ip=<CNTR_ADDR> \
--set csiSecret.simplybk.secret=<CLUSTER_SECRET> \
--set logicalVolume.pool_name=<POOL_NAME> \
--set storagenode.create=true
demo@demo ~> export CLUSTER_UUID="4502977c-ae2d-4046-a8c5-ccc7fa78eb9a"
demo@demo ~> export CLUSTER_SECRET="oal4PVNbZ80uhLMah2Bs"
demo@demo ~> export CNTR_ADDR="http://192.168.10.1/"
demo@demo ~> export POOL_NAME="test"
demo@demo ~> helm repo add simplyblock-csi https://install.simplyblock.io/helm
"simplyblock-csi" has been added to your repositories
demo@demo ~> helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "simplyblock-csi" chart repository
...Successfully got an update from the "kasten" chart repository
Update Complete. ⎈Happy Helming!⎈
demo@demo ~> helm install -n simplyblock-csi --create-namespace simplyblock-csi simplyblock-csi/spdk-csi \
--set csiConfig.simplybk.uuid=${CLUSTER_UUID} \
--set csiConfig.simplybk.ip=${CNTR_ADDR} \
--set csiSecret.simplybk.secret=${CLUSTER_SECRET} \
--set logicalVolume.pool_name=${POOL_NAME}
NAME: simplyblock-csi
LAST DEPLOYED: Wed Mar 5 15:06:02 2025
NAMESPACE: simplyblock-csi
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Simplyblock SPDK Driver is getting deployed to your cluster.
To check CSI SPDK Driver pods status, please run:
kubectl --namespace=simplyblock-csi get pods --selector="release=simplyblock-csi" --watch
demo@demo ~> kubectl --namespace=simplyblock-csi get pods --selector="release=simplyblock-csi" --watch
NAME READY STATUS RESTARTS AGE
spdkcsi-controller-0 6/6 Running 0 30s
spdkcsi-node-tzclt 2/2 Running 0 30s