Hybrid Setup
Network Preparation
Simplyblock recommends two individual network interfaces, one for the control plane and one for the storage plane. Hence, in the following installation description, we assume two separated subnets. To install simplyblock in your environment, you may have to adopt these commands to match your configuration.
| Network interface | Network definition | Abbreviation | Subnet |
|---|---|---|---|
| eth0 | Control Plane | control | 192.168.10.0/24 |
| eth1 | Storage Plane | storage | 10.10.10.0/24 |
Storage Plane Installation
Firewall Configuration (SP)
Control Plane Installation
The first step when installing simplyblock, is to install the control plane. The control plane manages one or more storage clusters. If an existing control plane is available and the new cluster should be added to it, this section can be skipped. Jump right to the Storage Plane Installation.
Firewall Configuration (CP)
Simplyblock requires a number of TCP and UDP ports to be opened from certain networks. Additionally, it requires IPv6 to be disabled on management nodes.
Following is a list of all ports (TCP and UDP) required for operation as a management node. Attention is required, as this list is for management nodes only. Storage nodes have a different port configuration. See the Firewall Configuration section for the storage plane.
| Service | Direction | Source / Target Network | Port | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| Cluster API | ingress | storage, control, admin | 80 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Graylog | ingress | storage, control | 12201 | TCP / UDP |
| Graylog | ingress | storage, control | 12202 | TCP |
| Graylog | ingress | storage, control | 13201 | TCP |
| Graylog | ingress | storage, control | 13202 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| FoundationDB | ingress | storage, control | 4500 | TCP |
| Prometheus | ingress | storage, control | 9100 | TCP |
| Cluster Control | egress | storage, control | 8080-8890 | TCP |
| spdk-http-proxy | egress | storage, control | 5000 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
With the previously defined subnets, the following snippet disables IPv6 and configures the iptables automatically.
Danger
The example assumes that you have an external firewall between the admin network and the public internet!
If this is not the case, ensure the correct source access for ports 22 and 80.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
# Clean up
sudo iptables -F SIMPLYBLOCK
sudo iptables -D DOCKER-FORWARD -j SIMPLYBLOCK
sudo iptables -X SIMPLYBLOCK
# Setup
sudo iptables -N SIMPLYBLOCK
sudo iptables -I DOCKER-FORWARD 1 -j SIMPLYBLOCK
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A SIMPLYBLOCK -m state --state ESTABLISHED,RELATED -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 80 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2375 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2377 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 4500 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 4789 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 9100 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 12201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 12201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 12202 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 13201 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 13202 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -s 0.0.0.0/0 -j DROP
Management Node Installation
Now that the network is configured, the management node software can be installed.
Simplyblock provides a command line interface called sbcli-pre. It's built in Python and required Python 3 and Pip (the
Python package manager) installed on the machine. This can be achieved with yum.
sudo yum -y install python3-pip
Afterward, the sbcli-pre command line interface can be installed. Upgrading the CLI later on, uses the same command.
sudo pip install sbcli-pre --upgrade
Recommendation
Simplyblock recommends to only upgrade sbcli-pre if a system upgrade is executed to prevent potential
incompatibilities between the running simplyblock cluster and the version of sbcli-pre.
At this point, a quick check with the simplyblock provided system check can reveal potential issues quickly.
curl -L https://sblk.xyz/prerequisites | bash
If the check succeeds, it's time to set up the primary management node:
sbcli-pre cluster create --ifname=<IF_NAME> --ha-type=ha
The output should look something like this:
[root@vm11 ~]# sbcli-pre cluster create --ifname=eth0 --ha-type=ha
2025-02-26 12:37:06,097: INFO: Installing dependencies...
2025-02-26 12:37:13,338: INFO: Installing dependencies > Done
2025-02-26 12:37:13,358: INFO: Node IP: 192.168.10.1
2025-02-26 12:37:13,510: INFO: Configuring docker swarm...
2025-02-26 12:37:14,199: INFO: Configuring docker swarm > Done
2025-02-26 12:37:14,200: INFO: Adding new cluster object
File moved to /usr/local/lib/python3.9/site-packages/simplyblock_core/scripts/alerting/alert_resources.yaml successfully.
2025-02-26 12:37:14,269: INFO: Deploying swarm stack ...
2025-02-26 12:38:52,601: INFO: Deploying swarm stack > Done
2025-02-26 12:38:52,604: INFO: deploying swarm stack succeeded
2025-02-26 12:38:52,605: INFO: Configuring DB...
2025-02-26 12:39:06,003: INFO: Configuring DB > Done
2025-02-26 12:39:06,106: INFO: Settings updated for existing indices.
2025-02-26 12:39:06,147: INFO: Template created for future indices.
2025-02-26 12:39:06,505: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "Cluster", "message": "Cluster created 7bef076c-82b7-46a5-9f30-8c938b30e655", "caused_by": "cli"}
2025-02-26 12:39:06,529: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "MgmtNode", "message": "Management node added vm11", "caused_by": "cli"}
2025-02-26 12:39:06,533: INFO: Done
2025-02-26 12:39:06,535: INFO: New Cluster has been created
2025-02-26 12:39:06,535: INFO: 7bef076c-82b7-46a5-9f30-8c938b30e655
7bef076c-82b7-46a5-9f30-8c938b30e655
If the deployment was successful, the last line returns the cluster id. This should be noted down. It's required in further steps of the installation.
Additionally to the cluster id, the cluster secret is required in many further steps. The following command can be used to retrieve it.
sbcli-pre cluster get-secret <CLUSTER_ID>
[root@vm11 ~]# sbcli-pre cluster get-secret 7bef076c-82b7-46a5-9f30-8c938b30e655
e8SQ1ElMm8Y9XIwyn8O0
Secondary Management Nodes
A production cluster, requires at least three management nodes in the control plane. Hence, additional management nodes need to be added.
On the secondary nodes, the network requires the same configuration as on the primary. Executing the commands under Firewall Configuration (CP) will get the node prepared.
Afterward, Python, Pip, and sbcli-pre need to be installed.
sudo yum -y install python3-pip
pip install sbcli-pre --upgrade
Finally, we deploy the management node software and join the control plane cluster.
sbcli-pre mgmt add <CP_PRIMARY_IP> <CLUSTER_ID> <CLUSTER_SECRET> <IF_NAME>
Running against the primary management node in the control plane should create an output similar to the following example:
[demo@demo ~]# sbcli-pre mgmt add 192.168.10.1 7bef076c-82b7-46a5-9f30-8c938b30e655 e8SQ1ElMm8Y9XIwyn8O0 eth0
2025-02-26 12:40:17,815: INFO: Cluster found, NQN:nqn.2023-02.io.simplyblock:7bef076c-82b7-46a5-9f30-8c938b30e655
2025-02-26 12:40:17,816: INFO: Installing dependencies...
2025-02-26 12:40:25,606: INFO: Installing dependencies > Done
2025-02-26 12:40:25,626: INFO: Node IP: 192.168.10.2
2025-02-26 12:40:26,802: INFO: Joining docker swarm...
2025-02-26 12:40:27,719: INFO: Joining docker swarm > Done
2025-02-26 12:40:32,726: INFO: Adding management node object
2025-02-26 12:40:32,745: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "OBJ_CREATED", "object_name": "MgmtNode", "message": "Management node added vm12", "caused_by": "cli"}
2025-02-26 12:40:32,752: INFO: Done
2025-02-26 12:40:32,755: INFO: Node joined the cluster
cdde125a-0bf3-4841-a6ef-a0b2f41b8245
From here, additional management nodes can be added to the control plane cluster. If the control plane cluster is ready, the storage plane can be installed.
Storage Plane Installation
The installation of a storage plane requires a functioning control plane. If no control plane cluster is available yet, it must be installed beforehand. Jump right to the Control Plane Installation.
The following examples assume two subnets to be available. These subnets are defined as shown in Network Preparation.
Firewall Configuration (SP)
Simplyblock requires a number of TCP and UDP ports to be opened from certain networks. Additionally, it requires IPv6 to be disabled on management nodes.
Following is a list of all ports (TCP and UDP) required for operation as a storage node. Attention is required, as this list is for storage nodes only. Management nodes have a different port configuration. See the Firewall Configuration section for the control plane.
| Service | Direction | Source / Target Network | Port(s) | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| bdts | ingress | storage | 4420 | TCP |
| Cluster Control | ingress | control | 5000 | TCP |
| spdk-http-proxy | ingress | storage, control | 8080-8890 | TCP |
| lvol-proxy | ingress | storage, control | 9090-9900 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| FoundationDB | egress | storage | 4500 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
| Graylog | egress | control | 12202 | TCP |
With the previously defined subnets, the following snippet disables IPv6 and configures the iptables automatically.
Danger
The example assumes that you have an external firewall between the admin network and the public internet!
If this is not the case, ensure the correct source access for port 22.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Docker Swarm, by default creates iptables entries open to the world. If no external firewall is available, the created iptables configuration needs to be restricted.
The following script will create additional iptables rules prepended to Docker's forwarding rules and only enabling access from internal networks. This script should be stored to /usr/local/sbin/simplyblock-iptables.sh.
#!/usr/bin/env bash
# Clean up
sudo iptables -F SIMPLYBLOCK
sudo iptables -D DOCKER-FORWARD -j SIMPLYBLOCK
sudo iptables -X SIMPLYBLOCK
# Setup
sudo iptables -N SIMPLYBLOCK
sudo iptables -I DOCKER-FORWARD 1 -j SIMPLYBLOCK
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2375 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2377 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 4420 -s 10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 4789 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 5000 -s 192.168.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 8080:8890 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 9090-9900 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -s 0.0.0.0/0 -j DROP
To automatically run this script whenever Docker is started or restarted, it must be attached to a Systemd service, stored as /etc/systemd/system/simplyblock-iptables.service.
[Unit]
Description=Simplyblock Iptables Restrictions for Docker
After=docker.service
BindsTo=docker.service
ReloadPropagatedFrom=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/simplyblock-iptables.sh
ExecReload=/usr/local/sbin/simplyblock-iptables.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
After both files are stored in their respective locations, the bash script needs to be made executable and the Systemd service needs to be enabled to start automatically.
chmod +x /usr/local/sbin/simplyblock-iptables.sh
systemctl enable simplyblock-iptables.service
systemctl start simplyblock-iptables.service
Storage Node Installation
Now that the network is configured, the storage node software can be installed.
Info
All storage nodes can be prepared at this point, as they are added to the cluster in the next step. Therefore, it is recommended to execute this step on all storage nodes, before moving to the next step.
Simplyblock provides a command line interface called sbcli-pre. It's built in Python and required Python 3 and Pip
(the Python package manager) installed on the machine. This can be achieved with yum.
sudo yum -y install python3-pip
Afterward, the sbcli-pre command line interface can be installed. Upgrading the CLI later on, uses the same command.
sudo pip install sbcli-pre --upgrade
Recommendation
Simplyblock recommends to only upgrade sbcli-pre if a system upgrade is executed to prevent potential
incompatibilities between the running simplyblock cluster and the version of sbcli-pre.
At this point, a quick check with the simplyblock provided system check can reveal potential issues quickly.
curl -L https://sblk.xyz/prerequisites | bash
Once the check is complete, the NVMe devices in each storage node can be prepared. To prevent data loss in case of a sudden power outage, NVMe devices need to be formatted for a specific LBA format.
Danger
Failing to format NVMe devices with the correct LBA format can lead to data loss or data corruption in the case of a sudden power outage or other loss of power.
The lsblk is the best way to find all NVMe devices attached to a system.
[demo@demo-3 ~]# sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 30G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 29G 0 part
├─rl-root 253:0 0 26G 0 lvm /
└─rl-swap 253:1 0 3G 0 lvm [SWAP]
nvme3n1 259:0 0 6.5G 0 disk
nvme2n1 259:1 0 70G 0 disk
nvme1n1 259:2 0 70G 0 disk
nvme0n1 259:3 0 70G 0 disk
In the example, we see four NVMe devices. Three devices of 70GiB and one device with 6.5GiB storage capacity.
To find the correct LBA format (lbaf) for each of the devices, the nvme cli can be used.
sudo nvme id-ns /dev/nvmeXnY
The output depends on the NVMe device itself, but looks something like this:
[demo@demo-3 ~]# sudo nvme id-ns /dev/nvme0n1
NVME Identify Namespace 1:
...
lbaf 0 : ms:0 lbads:9 rp:0
lbaf 1 : ms:8 lbads:9 rp:0
lbaf 2 : ms:16 lbads:9 rp:0
lbaf 3 : ms:64 lbads:9 rp:0
lbaf 4 : ms:0 lbads:12 rp:0 (in use)
lbaf 5 : ms:8 lbads:12 rp:0
lbaf 6 : ms:16 lbads:12 rp:0
lbaf 7 : ms:64 lbads:12 rp:0
From this output, the required lbaf configuration can be found. The necessary configuration has to have the following values:
| Property | Value |
|---|---|
| ms | 0 |
| lbads | 12 |
| rp | 0 |
In the example, the required LBA format is 4. If a NVMe device doesn't have that combination, any other lbads=12 combination will work. However, simplyblock recommends to ask for the best available combination.
In our example, the device is already formatted with the correct lbaf (see the "in use"). It is, however, recommended to always format the device before use.
To format the drive, the nvme cli is used again.
sudo nvme format --lbaf=<lbaf> --ses=0 /dev/nvmeXnY
The output of the command should give a successful response when executing similar to the below example.
[demo@demo-3 ~]# sudo nvme format --lbaf=4 --ses=0 /dev/nvme0n1
You are about to format nvme0n1, namespace 0x1.
WARNING: Format may irrevocably delete this device's data.
You have 10 seconds to press Ctrl-C to cancel this operation.
Use the force [--force] option to suppress this warning.
Sending format operation ...
Success formatting namespace:1
Simplyblock is built upon the NVMe over Fabrics standard and uses NVMe over TCP (NVMe/TCP) by default.
While the driver is part of the Linux kernel with kernel versions 5.x and later, it is not enabled by default. Hence, when using simplyblock, the driver needs to be loaded.
modprobe nvme-tcp
When loading the NVMe/TCP driver, the NVMe over Fabrics driver automatically get loaded to, as the former depends on its provided foundations.
It is possible to check for successful loading of both drivers with the following command:
lsmod | grep 'nvme_'
The response should list the drivers as nvme_tcp and nvme_fabrics as seen in the following example:
[demo@demo ~]# lsmod | grep 'nvme_'
nvme_tcp 57344 0
nvme_keyring 16384 1 nvme_tcp
nvme_fabrics 45056 1 nvme_tcp
nvme_core 237568 3 nvme_tcp,nvme,nvme_fabrics
nvme_auth 28672 1 nvme_core
t10_pi 20480 2 sd_mod,nvme_core
To make the driver loading persistent and survive system reboots, it has to be configured to be loaded at system startup time. This can be achieved by either adding it to /etc/modules (Debian / Ubuntu) or creating a config file under /etc/modules-load.d/ (Red Hat / Alma / Rocky).
echo "nvme-tcp" | sudo tee -a /etc/modules-load.d/nvme-tcp.conf
echo "nvme-tcp" | sudo tee -a /etc/modules
After rebooting the system, the driver should be loaded automatically. It can be checked again via the above provided
lsmod command.
With all NVMe devices prepared and the NVMe/TCP driver loaded, the storage node software can be deployed.
sudo sbcli-pre sn deploy --ifname eth0
The output will look something like the following example:
[demo@demo-3 ~]# sudo sbcli-pre sn deploy --ifname eth0
2025-02-26 13:35:06,991: INFO: NVMe SSD devices found on node:
2025-02-26 13:35:07,038: INFO: Installing dependencies...
2025-02-26 13:35:13,508: INFO: Node IP: 192.168.10.2
2025-02-26 13:35:13,623: INFO: Pulling image public.ecr.aws/simply-block/simplyblock:hmdi
2025-02-26 13:35:15,219: INFO: Recreating SNodeAPI container
2025-02-26 13:35:15,543: INFO: Pulling image public.ecr.aws/simply-block/ultra:main-latest
192.168.10.2:5000
On a successful deployment, the last line will provide the storage node's control channel address. This should be noted for all storage nodes, as it is required in the next step to attach the storage node to the simplyblock storage cluster.
Secondary Node Installation
A secondary node is a storage node without additional storage disks to contribute to the distributed storage pool. Apart from that, it is the same as a normal storage node.
However, due to the missing storage devices, preparing a secondary node only requires the NVMe/TCP driver to be loaded and the storage node software first-stage to be deployed.
Simplyblock is built upon the NVMe over Fabrics standard and uses NVMe over TCP (NVMe/TCP) by default.
While the driver is part of the Linux kernel with kernel versions 5.x and later, it is not enabled by default. Hence, when using simplyblock, the driver needs to be loaded.
modprobe nvme-tcp
When loading the NVMe/TCP driver, the NVMe over Fabrics driver automatically get loaded to, as the former depends on its provided foundations.
It is possible to check for successful loading of both drivers with the following command:
lsmod | grep 'nvme_'
The response should list the drivers as nvme_tcp and nvme_fabrics as seen in the following example:
[demo@demo ~]# lsmod | grep 'nvme_'
nvme_tcp 57344 0
nvme_keyring 16384 1 nvme_tcp
nvme_fabrics 45056 1 nvme_tcp
nvme_core 237568 3 nvme_tcp,nvme,nvme_fabrics
nvme_auth 28672 1 nvme_core
t10_pi 20480 2 sd_mod,nvme_core
To make the driver loading persistent and survive system reboots, it has to be configured to be loaded at system startup time. This can be achieved by either adding it to /etc/modules (Debian / Ubuntu) or creating a config file under /etc/modules-load.d/ (Red Hat / Alma / Rocky).
echo "nvme-tcp" | sudo tee -a /etc/modules-load.d/nvme-tcp.conf
echo "nvme-tcp" | sudo tee -a /etc/modules
After rebooting the system, the driver should be loaded automatically. It can be checked again via the above provided
lsmod command.
To deploy the first stage of the storage node, the following command must be executed.
sudo sbcli-pre sn deploy --ifname eth0
The output will look something like the following example:
[demo@demo-4 ~]# sudo sbcli-pre sn deploy --ifname eth0
2025-02-26 13:35:06,991: INFO: NVMe SSD devices found on node:
2025-02-26 13:35:07,038: INFO: Installing dependencies...
2025-02-26 13:35:13,508: INFO: Node IP: 192.168.10.4
2025-02-26 13:35:13,623: INFO: Pulling image public.ecr.aws/simply-block/simplyblock:hmdi
2025-02-26 13:35:15,219: INFO: Recreating SNodeAPI container
2025-02-26 13:35:15,543: INFO: Pulling image public.ecr.aws/simply-block/ultra:main-latest
192.168.10.4:5000
On a successful deployment, the last line will provide the secondary node's control channel address. This should be noted, as it is required in the next step to attach the secondary node to the simplyblock storage cluster.
Attach the Storage Node to the Control Plane
When all storage nodes are prepared, they can be added to the storage cluster.
Warning
The following command are executed from a management node. Attaching a storage node to a control plane is executed from a management node.
sudo sbcli-pre sn add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
--max-lvol <MAX_LOGICAL_VOLUMES> \
--max-prov <MAX_PROVISIONING_CAPACITY> \
--number-of-devices <NUM_STOR_NVME> \
--partitions <NUM_OF_PARTITIONS> \
--data-nics <DATA_IF>
Info
The number of partitions (<NUM_OF_PARTITIONS>) depends on the storage node setup. If a storage node has a separate journaling device (which is strongly recommended), the value should be zero (0) to prevent the storage devices to be partitioned. This improves the performance and prevents device sharing between the journal and actual data storage location.
The output will look something like the following example:
[demo@demo ~]# sudo sbcli-pre sn add-node 7bef076c-82b7-46a5-9f30-8c938b30e655 192.168.10.2:5000 eth0 --max-lvol 50 --max-prov 500g --number-of-devices 3 --partitions 0 --data-nics eth1
2025-02-26 14:55:17,236: INFO: Adding Storage node: 192.168.10.2:5000
2025-02-26 14:55:17,340: INFO: Instance id: 0b0c825e-3d16-4d91-a237-51e55c6ffefe
2025-02-26 14:55:17,341: INFO: Instance cloud: None
2025-02-26 14:55:17,341: INFO: Instance type: None
2025-02-26 14:55:17,342: INFO: Instance privateIp: 192.168.10.2
2025-02-26 14:55:17,342: INFO: Instance public_ip: 192.168.10.2
2025-02-26 14:55:17,347: INFO: Node Memory info
2025-02-26 14:55:17,347: INFO: Total: 24.3 GB
2025-02-26 14:55:17,348: INFO: Free: 23.2 GB
2025-02-26 14:55:17,348: INFO: Minimum required huge pages memory is : 14.8 GB
2025-02-26 14:55:17,349: INFO: Joining docker swarm...
2025-02-26 14:55:21,060: INFO: Deploying SPDK
2025-02-26 14:55:31,969: INFO: adding alceml_2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4
2025-02-26 14:55:32,010: INFO: creating subsystem nqn.2023-02.io.simplyblock:vm12:dev:2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4
2025-02-26 14:55:32,022: INFO: adding listener for nqn.2023-02.io.simplyblock:vm12:dev:2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4 on IP 10.10.10.2
2025-02-26 14:55:32,303: INFO: Connecting to remote devices
2025-02-26 14:55:32,321: INFO: Connecting to remote JMs
2025-02-26 14:55:32,342: INFO: Make other nodes connect to the new devices
2025-02-26 14:55:32,346: INFO: Setting node status to Active
2025-02-26 14:55:32,357: INFO: {"cluster_id": "3196b77c-e6ee-46c3-8291-736debfe2472", "event": "STATUS_CHANGE", "object_name": "StorageNode", "message": "Storage node status changed from: in_creation to: online", "caused_by": "monitor"}
2025-02-26 14:55:32,361: INFO: Sending event updates, node: 37b404b9-36aa-40b3-8b74-7f3af86bd5a5, status: online
2025-02-26 14:55:32,368: INFO: Sending to: 37b404b9-36aa-40b3-8b74-7f3af86bd5a5
2025-02-26 14:55:32,389: INFO: Connecting to remote devices
2025-02-26 14:55:32,442: WARNING: The cluster status is not active (unready), adding the node without distribs and lvstore
2025-02-26 14:55:32,443: INFO: Done
Repeat this process for all prepared storage nodes to add them to the storage plane.
Attach the Secondary Node to the Control Plane
Afterward, the secondary node needs to be added to the cluster.
sudo sbcli-pre sn add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
--data-nics <DATA_IF>
--is-secondary-node
The output will look something like the following example:
[demo@demo ~]# sudo sbcli-pre sn add-node 7bef076c-82b7-46a5-9f30-8c938b30e655 192.168.10.5:5000 ens18 --data-nics=ens16 --is-secondary-node
2025-02-28 13:34:57,877: INFO: Adding Storage node: 192.168.10.115:5000
2025-02-28 13:34:57,952: INFO: Node found: vm5
2025-02-28 13:34:57,953: INFO: Instance id: 5d679365-1361-40b0-bac0-3de949057bbc
2025-02-28 13:34:57,953: INFO: Instance cloud: None
2025-02-28 13:34:57,954: INFO: Instance type: None
2025-02-28 13:34:57,954: INFO: Instance privateIp: 192.168.10.5
2025-02-28 13:34:57,955: INFO: Instance public_ip: 192.168.10.5
2025-02-28 13:34:57,977: WARNING: Unsupported instance-type None for deployment
2025-02-28 13:34:57,977: INFO: Node Memory info
...
025-02-28 13:35:08,068: INFO: Connecting to remote devices
2025-02-28 13:35:08,111: INFO: Connecting to node 2f4dafb1-d610-42a7-9a53-13732459523e
2025-02-28 13:35:08,111: INFO: bdev found remote_alceml_378cf3b5-1959-4415-87bf-392fa1bbed6c_qosn1
2025-02-28 13:35:08,112: INFO: bdev found remote_alceml_c4c4011a-8f82-4c9d-8349-b4023a20b87c_qosn1
2025-02-28 13:35:08,112: INFO: bdev found remote_alceml_d27388b9-bbd8-4e82-8880-d8811aa45383_qosn1
2025-02-28 13:35:08,113: INFO: Connecting to node b7db725a-96e2-40d1-b41b-738495d97093
2025-02-28 13:35:08,113: INFO: bdev found remote_alceml_7f5ade89-53c6-440b-9614-ec24db3afbd9_qosn1
2025-02-28 13:35:08,114: INFO: bdev found remote_alceml_8d160125-f095-43ae-9781-16d841ae9719_qosn1
2025-02-28 13:35:08,114: INFO: bdev found remote_alceml_b0691372-1a4b-4fa9-a805-c2c1f311541c_qosn1
2025-02-28 13:35:08,114: INFO: Connecting to node 43560b0a-f966-405f-b27a-2c571a2bb4eb
2025-02-28 13:35:08,115: INFO: bdev found remote_alceml_29e74188-5efa-47d9-9282-84b4e46b77db_qosn1
2025-02-28 13:35:08,115: INFO: bdev found remote_alceml_a1efcbbf-328c-4f86-859f-fcfceae1c7a8_qosn1
2025-02-28 13:35:08,116: INFO: bdev found remote_alceml_cf2d3d24-e244-4a45-a71d-6383db07806f_qosn1
2025-02-28 13:35:08,274: WARNING: The cluster status is not active (unready), adding the node without distribs and lvstore
2025-02-28 13:35:08,274: INFO: Done
On a successful response, it's finally time to activate the storage plane.
Activate the Storage Cluster
The last step after all nodes are added to the storage cluster, the storage plane can be activated.
sudo sbcli-pre cluster activate <CLUSTER_ID>
The command output should look like this and respond with a successful activation of the storage cluster
[demo@demo ~]# sbcli-pre cluster activate 7bef076c-82b7-46a5-9f30-8c938b30e655
2025-02-28 13:35:26,053: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "STATUS_CHANGE", "object_name": "Cluster", "message": "Cluster status changed from unready to in_activation", "caused_by": "cli"}
2025-02-28 13:35:26,322: INFO: Connecting remote_jm_43560b0a-f966-405f-b27a-2c571a2bb4eb to 2f4dafb1-d610-42a7-9a53-13732459523e
2025-02-28 13:35:31,133: INFO: Connecting remote_jm_43560b0a-f966-405f-b27a-2c571a2bb4eb to b7db725a-96e2-40d1-b41b-738495d97093
2025-02-28 13:35:55,791: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "STATUS_CHANGE", "object_name": "Cluster", "message": "Cluster status changed from in_activation to active", "caused_by": "cli"}
2025-02-28 13:35:55,794: INFO: Cluster activated successfully
Hyper-Converged Storage Node Installation
Simplyblock is built upon the NVMe over Fabrics standard and uses NVMe over TCP (NVMe/TCP) by default.
While the driver is part of the Linux kernel with kernel versions 5.x and later, it is not enabled by default. Hence, when using simplyblock, the driver needs to be loaded.
modprobe nvme-tcp
When loading the NVMe/TCP driver, the NVMe over Fabrics driver automatically get loaded to, as the former depends on its provided foundations.
It is possible to check for successful loading of both drivers with the following command:
lsmod | grep 'nvme_'
The response should list the drivers as nvme_tcp and nvme_fabrics as seen in the following example:
[demo@demo ~]# lsmod | grep 'nvme_'
nvme_tcp 57344 0
nvme_keyring 16384 1 nvme_tcp
nvme_fabrics 45056 1 nvme_tcp
nvme_core 237568 3 nvme_tcp,nvme,nvme_fabrics
nvme_auth 28672 1 nvme_core
t10_pi 20480 2 sd_mod,nvme_core
To make the driver loading persistent and survive system reboots, it has to be configured to be loaded at system startup time. This can be achieved by either adding it to /etc/modules (Debian / Ubuntu) or creating a config file under /etc/modules-load.d/ (Red Hat / Alma / Rocky).
echo "nvme-tcp" | sudo tee -a /etc/modules-load.d/nvme-tcp.conf
echo "nvme-tcp" | sudo tee -a /etc/modules
After rebooting the system, the driver should be loaded automatically. It can be checked again via the above provided
lsmod command.
To install the simplyblock in Kubernetes, a helm chart is provided. While it can be installed manually, the helm chart is strongly recommended. The installation requires a few values to be available.
First we need the unique cluster id. Note down the cluster uuid of the cluster to access.
sudo sbcli-pre cluster list
An example of the output is below.
[demo@demo ~]# sbcli-pre cluster list
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
| UUID | NQN | ha_type | tls | mgmt nodes | storage nodes | Mod | Status |
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
| 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a | nqn.2023-02.io.simplyblock:4502977c-ae2d-4046-a8c5-ccc7fa78eb9a | ha | False | 1 | 4 | 1x1 | active |
+--------------------------------------+-----------------------------------------------------------------+---------+-------+------------+---------------+-----+--------+
In addition, we need the cluster secret. Note down the cluster secret.
sbcli-pre cluster get-secret <CLUSTER_UUID>
Retrieving the cluster secret will look somewhat like that.
[demo@demo ~]# sbcli-pre cluster get-secret 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a
oal4PVNbZ80uhLMah2Bs
Additionally, a storage pool is required. If a pool already exists, it can be reused. Otherwise, creating a storage pool can be created as following:
sbcli-pre pool add <POOL_NAME> <CLUSTER_UUID>
The last line of a successful storage pool creation returns the new pool id.
[demo@demo ~]# sbcli-pre pool add test 4502977c-ae2d-4046-a8c5-ccc7fa78eb9a
2025-03-05 06:36:06,093: INFO: Adding pool
2025-03-05 06:36:06,098: INFO: {"cluster_id": "4502977c-ae2d-4046-a8c5-ccc7fa78eb9a", "event": "OBJ_CREATED", "object_name": "Pool", "message": "Pool created test", "caused_by": "cli"}
2025-03-05 06:36:06,100: INFO: Done
ad35b7bb-7703-4d38-884f-d8e56ffdafc6 # <- Pool Id
The last item necessary before deploying simplyblock is the control plane address. This is any of the API addresses of a
management node. Meaning, if the primary management node has the IP of 192.168.10.1, the control plane address is
http://192.168.0.1. It is, however, recommended to front all management nodes, with a load balancing proxy, such as
HAproxy. In the latter case, the load balancer URL would be the address of the control plane.
Anyhow, deploying simplyblock using the provided helm chart comes down to providing the four necessary values, adding the helm chart repository, and installing the driver. In addition to the storage nodes, this will also install the Simplyblock CSI driver for seamless integration with the Kubernetes CSI persistent storage subsystem.
CLUSTER_UUID="<UUID>"
CLUSTER_SECRET="<SECRET>"
CNTR_ADDR="<CONTROL-PLANE-ADDR>"
POOL_NAME="<POOL-NAME>"
helm repo add simplyblock-csi https://install.simplyblock.io/helm
helm repo update
helm install -n simplyblock-csi \
--create-namespace simplyblock-csi \
simplyblock-csi/spdk-csi \
--set csiConfig.simplybk.uuid=<CLUSTER_UUID> \
--set csiConfig.simplybk.ip=<CNTR_ADDR> \
--set csiSecret.simplybk.secret=<CLUSTER_SECRET> \
--set logicalVolume.pool_name=<POOL_NAME> \
--set storagenode.create=true
demo@demo ~> export CLUSTER_UUID="4502977c-ae2d-4046-a8c5-ccc7fa78eb9a"
demo@demo ~> export CLUSTER_SECRET="oal4PVNbZ80uhLMah2Bs"
demo@demo ~> export CNTR_ADDR="http://192.168.10.1/"
demo@demo ~> export POOL_NAME="test"
demo@demo ~> helm repo add simplyblock-csi https://install.simplyblock.io/helm
"simplyblock-csi" has been added to your repositories
demo@demo ~> helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "simplyblock-csi" chart repository
...Successfully got an update from the "kasten" chart repository
Update Complete. ⎈Happy Helming!⎈
demo@demo ~> helm install -n simplyblock-csi --create-namespace simplyblock-csi simplyblock-csi/spdk-csi \
--set csiConfig.simplybk.uuid=${CLUSTER_UUID} \
--set csiConfig.simplybk.ip=${CNTR_ADDR} \
--set csiSecret.simplybk.secret=${CLUSTER_SECRET} \
--set logicalVolume.pool_name=${POOL_NAME}
NAME: simplyblock-csi
LAST DEPLOYED: Wed Mar 5 15:06:02 2025
NAMESPACE: simplyblock-csi
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Simplyblock SPDK Driver is getting deployed to your cluster.
To check CSI SPDK Driver pods status, please run:
kubectl --namespace=simplyblock-csi get pods --selector="release=simplyblock-csi" --watch
demo@demo ~> kubectl --namespace=simplyblock-csi get pods --selector="release=simplyblock-csi" --watch
NAME READY STATUS RESTARTS AGE
spdkcsi-controller-0 6/6 Running 0 30s
spdkcsi-node-tzclt 2/2 Running 0 30s