Prerequisites
When installing simplyblock control planes and storage planes, a number of prerequisites are important to understand.
Simplyblock uses Docker Swarm for the control plane cluster. In case of a bare metal or virtualized installation, it will also use Docker Swarm for the storage plane. Hence, Docker has to be installed.
Furthermore, simplyblock requires the installation of the sbcli-pre command line tool. This tool is written in Python.
Therefore, Python (3.5 or later) has to be installed. Likewise, pip, the Python package manager, has to be installed
with version 20 or later.
To install sbcli-pre run:
sudo pip install sbcli-pre --upgrade
Node Sizing
Simplyblock has certain requirements in terms of CPU, RAM, and storage. See the specific Node Sizing documentation to learn more.
Node Instance Type
Simplyblock recommends pre-tested and verified instance types. Those instance types are:
- i3en.6xlarge
- i4i.8xlarge
Network Configuration
Simplyblock requires a number of network ports to be available from different networks. The configuration of the required network ports is provided in the installation documentation.
Additionally, IPv6 must be disabled on all nodes running simplyblock.
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
AWS Networks
Specifically for AWS, simplyblock strong advises to use individual networks for the control plane and storage plane.
For access to the Cluster Management API, simplyblock recommends using an AWS load balancer as a front instead of making the API available directly.
Warning
Amazon Linux 2 and Amazon Linux 2023 do not support NVMe over Fabrics Multipathing right now!
Careful with "Up To GBit/s Instances"
AWS heavily discounts instances with "up to GBit/s" network bandwidth. These instances use a credit-based system to calculate the currently available network bandwidth. These instances initially offer a good throughput until the credits are used up which happens very quickly.
In addition to the limitation on the maximum available bandwidth, AWS heavily limits the number of packets per second.
Recommendation
Simplyblock strongly recommends using EC2 instances with dedicated bandwidth. Generally recommended Amazon EC2 instances are i3en.6xlarge and i4i.8xlarge.
Network Ports for Control Plane
| Service | Direction | Source / Target Network | Port | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| Cluster API | ingress | storage, control, admin | 80 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Graylog | ingress | storage, control | 12201 | TCP / UDP |
| Graylog | ingress | storage, control | 12202 | TCP |
| Graylog | ingress | storage, control | 13201 | TCP |
| Graylog | ingress | storage, control | 13202 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| FoundationDB | ingress | storage, control | 4500 | TCP |
| Prometheus | ingress | storage, control | 9100 | TCP |
| Cluster Control | egress | storage, control | 8080-8890 | TCP |
| spdk-http-proxy | egress | storage, control | 5000 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
Network Ports for Storage Plane
| Service | Direction | Source / Target Network | Port(s) | Protocol(s) |
|---|---|---|---|---|
| ICMP | ingress | control | - | ICMP |
| bdts | ingress | storage | 4420 | TCP |
| Cluster Control | ingress | control | 5000 | TCP |
| spdk-http-proxy | ingress | storage, control | 8080-8890 | TCP |
| lvol-proxy | ingress | storage, control | 9090-9900 | TCP |
| SSH | ingress | storage, control, admin | 22 | TCP |
| Docker Daemon Remote Access | ingress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | ingress | storage, control | 2377 | TCP |
| Docker Overlay Network | ingress | storage, control | 4789 | UDP |
| Docker Network Discovery | ingress | storage, control | 7946 | TCP / UDP |
| FoundationDB | egress | storage | 4500 | TCP |
| Docker Daemon Remote Access | egress | storage, control | 2375 | TCP |
| Docker Swarm Remote Access | egress | storage, control | 2377 | TCP |
| Docker Overlay Network | egress | storage, control | 4789 | UDP |
| Docker Network Discovery | egress | storage, control | 7946 | TCP / UDP |
| Graylog | egress | control | 12202 | TCP |
Storage Configuration
Simplyblock has certain requirements in terms of storage. While the most important facts are provided in the installation section, here are things to consider.
Root Volume
The volume mounted as the root directory has to provide at least 35GiB of free capacity. More free space is recommended, especially for control plane nodes which collect logs and the cluster state.
NVMe Devices
NVMe devices used for simplyblock, should ALWAYS be formatted using the nvme command line tool before adding them
to a simplyblock storage node. Failing to do so can negatively impact storage performance and lead to data corruption
or even data loss in case of a sudden power outage.
The lsblk is the best way to find all NVMe devices attached to a system.
[demo@demo ~]# sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 30G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 29G 0 part
├─rl-root 253:0 0 26G 0 lvm /
└─rl-swap 253:1 0 3G 0 lvm [SWAP]
nvme3n1 259:0 0 6.5G 0 disk
nvme2n1 259:1 0 70G 0 disk
nvme1n1 259:2 0 70G 0 disk
nvme0n1 259:3 0 70G 0 disk
In the example, we see four NVMe devices. Three devices of 70GiB and one device with 6.5GiB storage capacity.
To find the correct LBA format (lbaf) for each of the devices, the nvme cli can be used.
sudo nvme id-ns /dev/nvmeXnY
The output depends on the NVMe device itself, but looks something like this:
[demo@demo ~]# sudo nvme id-ns /dev/nvme0n1
NVME Identify Namespace 1:
...
lbaf 0 : ms:0 lbads:9 rp:0
lbaf 1 : ms:8 lbads:9 rp:0
lbaf 2 : ms:16 lbads:9 rp:0
lbaf 3 : ms:64 lbads:9 rp:0
lbaf 4 : ms:0 lbads:12 rp:0 (in use)
lbaf 5 : ms:8 lbads:12 rp:0
lbaf 6 : ms:16 lbads:12 rp:0
lbaf 7 : ms:64 lbads:12 rp:0
From this output, the required lbaf configuration can be found. The necessary configuration has to have the following values:
| Property | Value |
|---|---|
| ms | 0 |
| lbads | 12 |
| rp | 0 |
In the example, the required LBA format is 4. If a NVMe device doesn't have that combination, any other lbads=12 combination will work. However, simplyblock recommends to ask for the best available combination.
In our example, the device is already formatted with the correct lbaf (see the "in use"). It is, however, recommended to always format the device before use.
To format the drive, the nvme cli is used again.
sudo nvme format --lbaf=<lbaf> --ses=0 /dev/nvmeXnY
The output of the command should give a successful response when executing similar to the below example.
[demo@demo ~]# sudo nvme format --lbaf=4 --ses=0 /dev/nvme0n1
You are about to format nvme0n1, namespace 0x1.
WARNING: Format may irrevocably delete this device's data.
You have 10 seconds to press Ctrl-C to cancel this operation.
Use the force [--force] option to suppress this warning.
Sending format operation ...
Success formatting namespace:1