Install Storage Plane
Storage Plane Installation
The installation of a storage plane requires a functioning control plane. If no control plane cluster is available yet, it must be installed beforehand. Jump right to the Control Plane Installation.
The following examples assume two subnets are available.
Firewall Configuration (SP)
Simplyblock requires a number of TCP and UDP ports to be opened from certain networks. Additionally, it requires IPv6 to be disabled on management nodes.
Following is a list of all ports (TCP and UDP) required for operation as a storage node. Attention is required, as this list is for storage nodes only. Management nodes have a different port configuration.
| Service | Direction | Source / Target Network | Port(s) | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| Storage node API | ingress | storage | 5000 | TCP |
| spdk-http-proxy | ingress | storage, control | 8080-8180 | TCP |
| hublvol-nvmf-subsys-port | ingress | storage, control | 9030-9059 | TCP |
| internal-nvmf-subsys-port | ingress | storage, control | 9060-9099 | TCP |
| lvol-nvmf-subsys-port | ingress | storage, control | 9100-9200 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| Greylog | ingress | control | 12202 | TCP |
| FoundationDB | egress | storage | 4500 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
| Greylog | egress | control | 12202 | TCP |
With the previously defined subnets, the following snippet disables IPv6 and configures the iptables automatically.
Danger
The example assumes that you have an external firewall between the admin network and the public internet!
If this is not the case, ensure the correct source access for port 22.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
Docker Swarm, by default, creates iptables entries open to the world. If no external firewall is available, the created iptables configuration needs to be restricted.
The following script will create additional iptables rules prepended to Docker's forwarding rules and only enabling access from internal networks. This script should be stored in /usr/local/sbin/simplyblock-iptables.sh.
#!/usr/bin/env bash
# Clean up
sudo iptables -F SIMPLYBLOCK
sudo iptables -D DOCKER-FORWARD -j SIMPLYBLOCK
sudo iptables -X SIMPLYBLOCK
# Setup
sudo iptables -N SIMPLYBLOCK
sudo iptables -I DOCKER-FORWARD 1 -j SIMPLYBLOCK
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2375 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 2377 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 4420 -s 10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 4789 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 5000 -s 192.168.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p udp --dport 7946 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 8080:8890 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -p tcp --dport 9090-9900 -s 192.168.10.0/24,10.10.10.0/24 -j RETURN
sudo iptables -A SIMPLYBLOCK -s 0.0.0.0/0 -j DROP
To automatically run this script whenever Docker is started or restarted, it must be attached to a Systemd service, stored as /etc/systemd/system/simplyblock-iptables.service.
[Unit]
Description=Simplyblock Iptables Restrictions for Docker
After=docker.service
BindsTo=docker.service
ReloadPropagatedFrom=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/simplyblock-iptables.sh
ExecReload=/usr/local/sbin/simplyblock-iptables.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
After both files are stored in their respective locations, the bash script needs to be made executable, and the Systemd service needs to be enabled to start automatically.
chmod +x /usr/local/sbin/simplyblock-iptables.sh
systemctl enable simplyblock-iptables.service
systemctl start simplyblock-iptables.service
Storage Node Installation
Now that the network is configured, the storage node software can be installed.
Info
All storage nodes can be prepared at this point, as they are added to the cluster in the next step. Therefore, it is recommended to execute this step on all storage nodes before moving to the next step.
Simplyblock provides a command line interface called sbctl. It's built in Python and requires
Python 3 and Pip (the Python package manager) are installed on the machine. This can be achieved with yum.
sudo yum -y install python3-pip
Afterward, the sbctl command line interface can be installed. Upgrading the CLI later on uses the
same command.
sudo pip install sbctl --upgrade
Recommendation
Simplyblock recommends to only upgrade sbctl if a system upgrade is executed to prevent potential
incompatibilities between the running simplyblock cluster and the version of sbctl.
At this point, a quick check with the simplyblock provided system check can reveal potential issues quickly.
curl -s -L https://install.simplyblock.io/scripts/prerequisites-sn.sh | bash
NVMe Device Preparation
Once the check is complete, the NVMe devices in each storage node can be prepared. To prevent data loss in case of a sudden power outage, NVMe devices need to be formatted for a specific LBA format.
Warning
Failing to format NVMe devices with the correct LBA format can lead to data loss or data corruption in the case of a sudden power outage or other loss of power. If you can't find the necessary LBA format, it is best to ask your simplyblock contact for further instructions.
On AWS, the necessary LBA format is not available. Simplyblock is, however, fully tested and supported with AWS.
The lsblk is the best way to find all NVMe devices attached to a system.
[demo@demo-3 ~]# sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 30G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 29G 0 part
├─rl-root 253:0 0 26G 0 lvm /
└─rl-swap 253:1 0 3G 0 lvm [SWAP]
nvme3n1 259:0 0 6.5G 0 disk
nvme2n1 259:1 0 70G 0 disk
nvme1n1 259:2 0 70G 0 disk
nvme0n1 259:3 0 70G 0 disk
In the example, we see four NVMe devices. Three devices of 70GiB and one device with 6.5GiB storage capacity.
To find the correct LBA format (lbaf) for each of the devices, the nvme CLI can be used.
sudo nvme id-ns /dev/nvmeXnY
The output depends on the NVMe device itself, but looks something like this:
[demo@demo-3 ~]# sudo nvme id-ns /dev/nvme0n1
NVME Identify Namespace 1:
...
lbaf 0 : ms:0 lbads:9 rp:0
lbaf 1 : ms:8 lbads:9 rp:0
lbaf 2 : ms:16 lbads:9 rp:0
lbaf 3 : ms:64 lbads:9 rp:0
lbaf 4 : ms:0 lbads:12 rp:0 (in use)
lbaf 5 : ms:8 lbads:12 rp:0
lbaf 6 : ms:16 lbads:12 rp:0
lbaf 7 : ms:64 lbads:12 rp:0
From this output, the required lbaf configuration can be found. The necessary configuration has to have the following values:
| Property | Value |
|---|---|
| ms | 0 |
| lbads | 12 |
| rp | 0 |
In the example, the required LBA format is 4. If an NVMe device doesn't have that combination, any other lbads=12 combination will work. However, simplyblock recommends asking for the best available combination.
Info
In some rare cases, no lbads=12 combination will be available. In this case, it is ok to leave the current setup. This is specifically true for certain cloud providers such as AWS.
In our example, the device is already formatted with the correct lbaf (see the "in use"). It is, however, recommended to always format the device before use.
To format the drive, the nvme cli is used again.
sudo nvme format --lbaf=<lbaf> --ses=0 /dev/nvmeXnY
The output of the command should give a successful response when executed similarly to the example below.
[demo@demo-3 ~]# sudo nvme format --lbaf=4 --ses=0 /dev/nvme0n1
You are about to format nvme0n1, namespace 0x1.
WARNING: Format may irrevocably delete this device's data.
You have 10 seconds to press Ctrl-C to cancel this operation.
Use the force [--force] option to suppress this warning.
Sending format operation ...
Success formatting namespace:1
Warning
This operation needs to be repeated for each NVMe device that will be handled by simplyblock.
Load the NVMe over Fabrics Kernel Modules
Simplyblock is built upon the NVMe over Fabrics standard and uses NVMe over TCP (NVMe/TCP) by default.
While the driver is part of the Linux kernel with kernel versions 5.x and later, it is not enabled by default. Hence, when using simplyblock, the driver needs to be loaded.
modprobe nvme-tcp
When loading the NVMe/TCP driver, the NVMe over Fabrics driver automatically get loaded to, as the former depends on its provided foundations.
It is possible to check for successful loading of both drivers with the following command:
lsmod | grep 'nvme_'
The response should list the drivers as nvme_tcp and nvme_fabrics as seen in the following example:
[demo@demo ~]# lsmod | grep 'nvme_'
nvme_tcp 57344 0
nvme_keyring 16384 1 nvme_tcp
nvme_fabrics 45056 1 nvme_tcp
nvme_core 237568 3 nvme_tcp,nvme,nvme_fabrics
nvme_auth 28672 1 nvme_core
t10_pi 20480 2 sd_mod,nvme_core
To make the driver loading persistent and survive system reboots, it has to be configured to be loaded at system startup time. This can be achieved by either adding it to /etc/modules (Debian / Ubuntu) or creating a config file under /etc/modules-load.d/ (Red Hat / Alma / Rocky).
echo "nvme-tcp" | sudo tee -a /etc/modules-load.d/nvme-tcp.conf
echo "nvme-tcp" | sudo tee -a /etc/modules
After rebooting the system, the driver should be loaded automatically. It can be checked again via the above provided
lsmod command.
Configuration and Deployment
With all NVMe devices prepared and the NVMe/TCP driver loaded, the storage node software can be deployed.
The actual deployment process happens in three steps: - Creating the storage node configuration - Deploy the first stage (the storage node API) - Deploy the second stage (add the storage node to the cluster), happening from a management node
The configuration process creates the configuration file, which contains all the assignments of NVMe devices, NICs, and potentially available NUMA nodes. By default, simplyblock will configure one storage node per NUMA node.
sudo sbctl storage-node configure \
--max-lvol <MAX_LOGICAL_VOLUMES> \
--max-size <MAX_PROVISIONING_CAPACITY>
[demo@demo-3 ~]# sudo sbctl sn configure --nodes-per-socket=2 --max-lvol=50 --max-size=1T
2025-05-14 10:40:17,460: INFO: 0000:00:04.0 is already bound to nvme.
0000:00:1e.0
0000:00:1e.0
0000:00:1f.0
0000:00:1f.0
0000:00:1e.0
0000:00:1f.0
2025-05-14 10:40:17,841: INFO: JSON file successfully written to /etc/simplyblock/sn_config_file
2025-05-14 10:40:17,905: INFO: JSON file successfully written to /etc/simplyblock/system_info
True
A full set of the parameters for the configure subcommand can be found in the CLI reference.
After the configuration has been created, the first stage deployment can be executed
sudo sbctl storage-node deploy --ifname eth0
The output will look something like the following example:
[demo@demo-3 ~]# sudo sbctl storage-node deploy --ifname eth0
2025-02-26 13:35:06,991: INFO: NVMe SSD devices found on node:
2025-02-26 13:35:07,038: INFO: Installing dependencies...
2025-02-26 13:35:13,508: INFO: Node IP: 192.168.10.2
2025-02-26 13:35:13,623: INFO: Pulling image public.ecr.aws/simply-block/simplyblock:hmdi
2025-02-26 13:35:15,219: INFO: Recreating SNodeAPI container
2025-02-26 13:35:15,543: INFO: Pulling image public.ecr.aws/simply-block/ultra:main-latest
192.168.10.2:5000
On a successful deployment, the last line will provide the storage node's control channel address. This should be noted for all storage nodes, as it is required in the next step to attach the storage node to the simplyblock storage cluster.
When all storage nodes are added, it's finally time to activate the storage plane.
Attach the Storage Node to the Control Plane
When all storage nodes are prepared, they can be added to the storage cluster.
Warning
The following commands are executed from a management node. Attaching a storage node to a control plane is executed from a management node.
sudo sbctl storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
--max-lvol <MAX_LOGICAL_VOLUMES> \
--max-prov <MAX_PROVISIONING_CAPACITY> \
--number-of-devices <NUM_STOR_NVME> \
--partitions <NUM_OF_PARTITIONS> \
--data-nics <DATA_IF>
Info
The number of partitions (NUM_OF_PARTITIONS) depends on the storage node setup. If a storage node has a separate journaling device (which is strongly recommended), the value should be zero (0) to prevent the storage devices from being partitioned. This improves the performance and prevents device sharing between the journal and the actual data storage location.
The output will look something like the following example:
[demo@demo ~]# sudo sbctl storage-node add-node 7bef076c-82b7-46a5-9f30-8c938b30e655 192.168.10.2:5000 eth0 --max-lvol 50 --max-prov 500g --number-of-devices 3 --partitions 0 --data-nics eth1
2025-02-26 14:55:17,236: INFO: Adding Storage node: 192.168.10.2:5000
2025-02-26 14:55:17,340: INFO: Instance id: 0b0c825e-3d16-4d91-a237-51e55c6ffefe
2025-02-26 14:55:17,341: INFO: Instance cloud: None
2025-02-26 14:55:17,341: INFO: Instance type: None
2025-02-26 14:55:17,342: INFO: Instance privateIp: 192.168.10.2
2025-02-26 14:55:17,342: INFO: Instance public_ip: 192.168.10.2
2025-02-26 14:55:17,347: INFO: Node Memory info
2025-02-26 14:55:17,347: INFO: Total: 24.3 GB
2025-02-26 14:55:17,348: INFO: Free: 23.2 GB
2025-02-26 14:55:17,348: INFO: Minimum required huge pages memory is : 14.8 GB
2025-02-26 14:55:17,349: INFO: Joining docker swarm...
2025-02-26 14:55:21,060: INFO: Deploying SPDK
2025-02-26 14:55:31,969: INFO: adding alceml_2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4
2025-02-26 14:55:32,010: INFO: creating subsystem nqn.2023-02.io.simplyblock:vm12:dev:2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4
2025-02-26 14:55:32,022: INFO: adding listener for nqn.2023-02.io.simplyblock:vm12:dev:2d1c235a-1f4d-44c7-9ac1-1db40e23a2c4 on IP 10.10.10.2
2025-02-26 14:55:32,303: INFO: Connecting to remote devices
2025-02-26 14:55:32,321: INFO: Connecting to remote JMs
2025-02-26 14:55:32,342: INFO: Make other nodes connect to the new devices
2025-02-26 14:55:32,346: INFO: Setting node status to Active
2025-02-26 14:55:32,357: INFO: {"cluster_id": "3196b77c-e6ee-46c3-8291-736debfe2472", "event": "STATUS_CHANGE", "object_name": "StorageNode", "message": "Storage node status changed from: in_creation to: online", "caused_by": "monitor"}
2025-02-26 14:55:32,361: INFO: Sending event updates, node: 37b404b9-36aa-40b3-8b74-7f3af86bd5a5, status: online
2025-02-26 14:55:32,368: INFO: Sending to: 37b404b9-36aa-40b3-8b74-7f3af86bd5a5
2025-02-26 14:55:32,389: INFO: Connecting to remote devices
2025-02-26 14:55:32,442: WARNING: The cluster status is not active (unready), adding the node without distribs and lvstore
2025-02-26 14:55:32,443: INFO: Done
Repeat this process for all prepared storage nodes to add them to the storage plane.
Activate the Storage Cluster
The last step, after all nodes are added to the storage cluster, is to activate the storage plane.
sudo sbctl cluster activate <CLUSTER_ID>
The command output should look like this, and respond with a successful activation of the storage cluster
[demo@demo ~]# sbctl cluster activate 7bef076c-82b7-46a5-9f30-8c938b30e655
2025-02-28 13:35:26,053: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "STATUS_CHANGE", "object_name": "Cluster", "message": "Cluster status changed from unready to in_activation", "caused_by": "cli"}
2025-02-28 13:35:26,322: INFO: Connecting remote_jm_43560b0a-f966-405f-b27a-2c571a2bb4eb to 2f4dafb1-d610-42a7-9a53-13732459523e
2025-02-28 13:35:31,133: INFO: Connecting remote_jm_43560b0a-f966-405f-b27a-2c571a2bb4eb to b7db725a-96e2-40d1-b41b-738495d97093
2025-02-28 13:35:55,791: INFO: {"cluster_id": "7bef076c-82b7-46a5-9f30-8c938b30e655", "event": "STATUS_CHANGE", "object_name": "Cluster", "message": "Cluster status changed from in_activation to active", "caused_by": "cli"}