Skip to content

Deploy with Linux Block Devices

Experimental

Deploying a storage plane on Linux block devices instead of NVMe PCIe devices follows the standard storage plane installation flow. Only the differing steps are described here. Background on the device mode, the eligibility rules, and the device identity is found under Linux Block Devices (lblk).

Warning

Linux block device support is experimental. It is intended for evaluation, for test environments, and for deployments where NVMe devices are unavailable, such as cloud instances with attached volumes only. For production-grade performance, local NVMe devices are recommended.

Cluster Creation

The device mode is a cluster-global, deploy-time choice made when the cluster is created on the control plane:

Creating a cluster in lblk device mode
sudo sbctl cluster create --device-mode lblk

Two values are accepted by --device-mode: nvme (the default) and lblk. The mode is followed by all storage nodes of the cluster and cannot be changed after creation.

Important

All control plane and storage node services must run a software version that supports the lblk device mode before a cluster is created with it.

Storage Node Configuration

On each storage node, devices are selected at storage-node configure time with the --lblk flag and an optional device selector:

Configuring a storage node with all eligible block devices
sudo sbctl storage-node configure --lblk --max-lvol <MAX_LOGICAL_VOLUMES>

Without a selector, every eligible disk on the host is used. An eligible disk is a whole, unmounted, unheld, and unpartitioned disk that is not the root disk (see the eligibility rules).

Devices can also be selected explicitly, by name or by serial number. The three selectors are mutually exclusive:

Selecting block devices by name
sudo sbctl storage-node configure \
    --lblk --blk-names sdb,sdc --max-subsystem 50
Selecting all eligible block devices except some
sudo sbctl storage-node configure \
    --lblk --blk-names-exclude sda --max-subsystem 50
Selecting block devices by serial number or WWN
sudo sbctl storage-node configure \
    --lblk --blk-serials S3EVNX0M602707,S3EVNX0M602708 --max-subsystem 50

A requested device that is busy (mounted, held, or otherwise ineligible) is an error. The configuration fails rather than silently skipping the device.

The selected devices are stored in the resulting configuration file (/etc/simplyblock/sn_config_file) with their name, serial, stable by-id path, size, and NUMA assignment. As in NVMe mode, the file can be reviewed and manually edited before deployment, for example to remove a device from the selection.

Partitioned Devices

A device carrying a partition table is not eligible by default. To reuse such a device, it is marked eligible with --force at configuration time:

Including a partitioned block device in the selection
sudo sbctl storage-node configure \
    --lblk --blk-names sdb --force --max-subsystem 50

The wipe itself happens later, at node addition, where it has to be requested explicitly with --force-format. Until then, no data is touched.

Storage Node Deployment and Addition

Node deployment is unchanged:

Deploying the storage node
sudo sbctl storage-node deploy --ifname eth0

Adding the node to the cluster from a control plane node is unchanged as well, apart from one additional flag. If partitioned devices were force-included at configuration time, with --force-format the node addition is instructed to wipe partition tables and filesystem signatures (wipefs) from those devices:

Adding the storage node while wiping partitioned devices
sudo sbctl storage-node add-node \
    --force-format <CLUSTER_ID> <NODE_IP>:5000 eth0

Danger

--force-format irreversibly destroys any data on the affected devices. The device selection in the node configuration has to be verified before the node is added.

During node addition, each selected device is wrapped in an SPDK AIO bdev. No kernel driver unbinding takes place, so the devices stay visible to the host OS but must not be used by anything else. The smallest device is used as the journal device, as in NVMe mode with journal-on-device deployments.

Everything after node addition (cluster activation, pool creation, volume provisioning, and client connection) is identical to an NVMe-mode cluster.

Verification

After activation, the devices are listed like NVMe devices, showing the device path instead of a PCIe address:

Listing the storage devices of a node
sudo sbctl storage-node list-devices <NODE_ID>

On the host, the devices remain kernel-owned and continue to be shown by lsblk. One aio_<serial> base bdev per device is exposed by the SPDK process.