26.3

Simplyblock is happy to release Simplyblock 26.3. The changes are grouped by the component they belong to: the simplyblock control plane and storage plane, and the Kubernetes components, meaning the Simplyblock Operator and Simplyblock CSI.

Simplyblock

The control plane and the storage plane are released together and share a version number. The changes below apply to every deployment, including a Kubernetes-based one.

New Features

Control Plane: Added asynchronous replication between two storage clusters. Several destinations per cluster are supported, the cadence and the retention are set per replication policy, and a failover is requested for a single volume, a whole policy, or a whole destination.
Control Plane: Added the planned cross-cluster cutover, by which a volume is handed over to another cluster under control. The remaining delta is shrunk, the source is fenced, and the client is moved to the target. The return to a recovered source runs as a delta, and the return to a fresh cluster as a full copy.
Control Plane: Added support for failure domains. Data chunks, parity chunks, journal copies, and the secondary and tertiary nodes are spread across distinct domains, with a host-disjoint fallback and a warning where that is not possible.
Control Plane: Added online storage node removal, which runs as a resumable background task and re-hosts the replicas and devices the node holds, and the expansion of an FTT 2 cluster by a single node. Several nodes are now added in parallel.
Control Plane: Added cluster-wide maintenance operations. Every node of a cluster is restarted in one step, and the creation, deletion, and modification of volumes, snapshots, clones, and pools is refused and resumed for the whole cluster. Read paths and the restarts, migrations, and repairs a cluster performs on itself are unaffected.
Control Plane: Added release-specific upgrade plug-ins, which run their preparation step before a cluster update and their completion step after it, and record the installed release on the cluster.
Control Plane: Added batch migration of namespaced volumes, by which every volume that shares an NVMe-oF subsystem is moved as one coordinated group. What a failed migration left behind on the target is removed idempotently.
Control Plane: Added quorum-based device exclusion. A device that draws I/O errors from a quorum of the other online nodes is excluded cluster-wide, independent of what its NVMe controller reports, which is the case a slow-but-connected device previously slipped through. A device whose latency deviates from the cluster average is reported as a warning event before it fails.
Control Plane: Added alerting from the cluster event log, which provisions Grafana rules that read the transition an entity made rather than the state it ended in. Journal events are now collected into the event log next to the distrib events.
Control Plane: Added a FoundationDB exporter with its own dashboard, request metrics of the management API, the device health information the storage plane holds, and a retention period of 30 days for cluster tasks and their logs.
Control Plane: Added a rebalance operation to the cluster API, validation of the erasure coding scheme by the management API, and independent NVMe device filters for a storage node, where the device model and the size range are accepted alone as well as together.
Storage Plane: The journal device of a node is now a RAID 0+1 stack instead of an N-way mirror across the journal partition of every drive. Write amplification is capped at two physical writes per journal record and per node, independent of the drive count, and a drive lifecycle event rebuilds the affected leg instead of the whole mirror.
Storage Plane: Added a second generation of distrib write protection. A new cluster is created on it, and an upgraded cluster is moved over once every online node has migrated. An offline node is brought over by its next restart.
Storage Plane: The ANA state change of a volume is now scoped to its namespace instead of the whole subsystem. A volume that shares its subsystem with other volumes is failed over, suspended, and migrated without moving their I/O to the other node.
Storage Plane: The core allocation of a storage node gives the distrib and poller threads their share before alceml, which takes its device-scaled count from what is left. Explicit layouts are added for hosts with two to five vCPUs.
General: Added diagnostics for a latency or crash investigation: the bdev latency histograms with a per-layer probe, by which a tail-latency spike is localized to a layer of the bdev stack, and the host directory /var/crash mounted into the storage plane container, so that a core dump survives the container.
General: Updated the packages inherited from the container base images and dropped wget from the image, to resolve open CVEs.

Fixes

Control Plane: Fixed the loss of I/O during a storage node restart, where an unbounded operation inside the port fence could outlast the client keep-alive.
Control Plane: Improved the recovery of a cluster after a full reboot. An activation that wedged is detected by a watchdog, automatic reactivation is gated on every node being online, and a suspended cluster is drained to all-offline before it is restarted.
Control Plane: Fixed storage nodes flapping to the down state on a health probe timeout, and a node that is down no longer counts toward the suspension of the cluster. The suspension is gated on an actual loss of the data plane rather than on an unreachable management network.
Control Plane: Fixed three ways in which a highly available volume could silently lose one of its paths, and the duplicate paths a repair left behind are now pruned. No leadership over an lvstore is granted while a restart or a recovery task owns the movement of it.
Control Plane: Fixed the failover of a namespaced volume, which the kernel rejected with mismatching identifiers.
Control Plane: Fixed the deletion pipeline of volumes and snapshots. A delete the cluster cannot complete is refused rather than accepted, a deferred delete completes once the node it waited for returns, and concurrent operations on one snapshot chain are mutually exclusive, which closes a corruption of the blob tree.
Control Plane: Fixed the migration of a volume whose snapshot chain spans several nodes, of an encrypted volume, and of a volume with a tertiary node. A migration whose source node goes offline is suspended instead of being cleaned up on the target.
Control Plane: Fixed the replication of the blob chain of a volume, which replicated only the snapshots of the volume itself. A failover only ever targets a fully replicated and live snapshot, and a replication no longer halts silently under back-pressure.
Control Plane: Fixed the addition of a storage node, which aborted on a lock timeout, on a stale in-creation record, or on an already provisioned device slot, and which did not survive the CPU topology reboot. A re-entry is idempotent now, and the status of a node, a device, a cluster, and a volume is written through a compare-and-set operation, so that a crashed writer cannot leave a stale status behind.
Control Plane: Fixed the capacity reporting, which mixed raw with effective capacity, and the cluster admission check, which did not count the utilization of snapshots. On a storage pool, the capacity check of a resize no longer counts the volume being resized twice.
Control Plane: Fixed the cloning of an encrypted volume, for which the crypto layer was missing from the recorded bdev stack, the key management of the local KMS, and the rendering of secrets in the management API.
Control Plane: Fixed a forced storage node shutdown that could block indefinitely, a cluster that grew by being reactivated, a cluster expansion that did not reattach the sibling failover of its donor node, and the leases of long-running tasks, which expired while the task was still running.
Control Plane: Improved the throughput of the snapshot and volume monitors, which process chains and nodes in parallel, and the logging to Graylog, which is written non-blocking, so that a slow log target no longer holds up the control plane.
Storage Plane: Improved the reliability of the graceful startup and the graceful shutdown of the storage plane. A storage plane whose RPC endpoint hangs on a live process is aborted instead of being waited for, and the wait for the journal replication of a peer is bounded, so that it cannot defer a restart indefinitely.
Storage Plane: Fixed the roles of the hub volumes, which are active on both sides within a node and passive between nodes, and the hub volume subsystem, which was lost across a restart.
Storage Plane: Fixed the re-admission of the devices of a node recovering from an outage, which was silently refused while the node had not yet reached the online state. The memory footprint of a node improved as well, since the small I/O buffers are sized from the subsystem count of the cluster.

New Features in 26.3.1

Control Plane: Added the alerts endpoint, GET /api/v2/clusters/<CLUSTER_UUID>/alerts/, which returns the conditions that currently need an operator.
Control Plane: Added hard per-object limits, enforced on every create and resize: 70 TiB per volume, 100 snapshots per volume, and 500 clones per snapshot. See Limits.
Control Plane: The shared NVMe-oF subsystems of a storage node are aligned with the storage pools, so that a subsystem carries the namespaces of exactly one pool.
Control Plane: Added the self-repair of a device left unavailable or removed, unless it was removed by the operator.
Control Plane: Added the Grafana rule API_request_latency_high, which fires when the mean request latency of the control plane API stays above 15 seconds.

Fixes in 26.3.1

Control Plane: Fixed the client I/O errors during the restart of a primary storage node, where a non-leader peer kept accepting I/O through a half-demoted lvstore.
Control Plane: Fixed the loss of the last usable path on a graceful shutdown, where the hublvol controller of a secondary or tertiary node was detached along with it.
Control Plane: Fixed the Invalid Namespace or Format error a client received from a subsystem that became reachable before its namespaces were attached.
Control Plane: Fixed the silent partial registration of the volume subsystems during an lvstore rebuild.
Control Plane: Fixed a volume deletion that stalled after its data had been freed, because the lvstore leadership moved while the asynchronous delete was running.
Control Plane: Fixed the journal geometry across an upgrade, where an existing journal could be rebuilt under the newer RAID layout.
Control Plane: Fixed the silent acceptance of a storage plane that was started without the NVMe options of the control plane.
Control Plane: Fixed the admission of a snapshot, which did not charge the utilized size of its source volume against the storage pool and the cluster.
Control Plane: Fixed the upgrade of a storage node whose record still carries the legacy lvstore_stack_secondary and lvstore_stack_tertiary values.
Control Plane: Fixed the flooding of the cluster event log with journal compression events, of which only a failure is persisted now.
Control Plane: Fixed the filesystem metrics of the node exporter, which were read from the wrong root path.
Control Plane: Fixed the handling of a FoundationDB backup, whose retention metadata was not recorded.
Control Plane: Fixed the log collection bundle, which aborted in Docker mode on a missing kubeconfig.
Control Plane: Fixed a fatal Python error at the shutdown of the storage node proxy, whose statistics thread outlived the interpreter.
Control Plane: Fixed the log levels of the management API and of the Docker Swarm services.

Kubernetes

The changes below apply to Kubernetes-based deployments only. They are delivered by the Simplyblock Operator and by Simplyblock CSI.

New Features

Operator: Added support for failure domains, by which erasure coding chunks and failover paths are spread across racks or availability zones. Workers are assigned to a domain through the StorageNodeSet, and the effective domain of a worker is reflected in the status of the StorageNode.
Operator: Added support for several StorageNodeSet resources per storage cluster, by which workers with different hardware or a different role are enrolled with their own configuration, and for the parallel addition of storage nodes.
Operator: Added live volume migration through the new VolumeMigration resource, by which the backing logical volume is moved between storage nodes while the volume stays online. The new NVMe-oF paths on the target node are validated on every consuming worker before the migration is continued.
Operator: Added the storage node drain, removal, and host migration workflows, requested with a StorageNodeOps resource. A removal suspends the node, migrates its volumes to the remaining nodes, verifies the result, and removes the node, and a migration relocates the identity of a storage node to a different worker.
Operator: Added support for asynchronous replication between two storage clusters, declared with the new ReplicationPair, ReplicationPolicy, and ReplicationSlot resources. Failover and failback are requested with a ReplicationOps resource, scoped to a replication target, a policy, or a single volume, and a PVC is enrolled into a policy through an annotation.
Operator: Added automatic, latency-based volume rebalancing and back storage data realignment, which are configured per cluster and triggered independently of each other.
Operator: Added support for topology-aware provisioning across several storage clusters, by which a volume is provisioned in the cluster that matches the topology of the consuming workload, the option to skip the creation of the FoundationDB cluster, and the adoption of an existing Helm-deployed storage cluster, storage pool, and storage nodes into the operator resources.
Operator: Added configuration of the image pull policy, the resource limits, and the SPDK system memory of the storage node workloads, and of a custom OpenShift MachineConfigPool, so that storage nodes are rebooted through a pool that is separate from the remaining workers. Every image reference is validated against the trusted registries, and a field that cannot be changed after creation is rejected instead of being ignored silently.
Operator: The Thanos data of the monitoring stack is now stored in MinIO instead of a filesystem-backed object store, and Prometheus can be deployed with more than one replica.
CSI Driver: Added support for co-locating the primary storage node of a volume with the worker that runs the consuming pod, and for the migration of namespaced volumes together with the coordinated restart of all pods that share an NVMe subsystem.
CSI Driver: Added the detection and the repair of stale NVMe-oF subsystems and controllers on a worker.
CSI Driver: The presence of a device is registered when the volume is connected, instead of only by polling, and the host NQN and the DH-CHAP secrets are forwarded at that point.

Fixes

Operator: Fixed the release of the NVMe target paths of a migration that never reaches its cutover, and the rebuild of the storage node configuration after an operator restart with an in-flight storage node migration.
Operator: Fixed the StoragePool resource remaining in Terminating when its backend UUID was never set, and the PVC validator, which did not accept all of the volume placement annotations.
CSI Driver: Fixed incompatibilities when formatting and mounting ext4 and XFS volumes, and the connection of a volume in a cluster with more than one storage pool.
CSI Driver: Fixed the loss of the kernel module configuration on the host, which was written into the ephemeral filesystem of the storage node container and was gone after the CPU topology reboot during a node addition.
CSI Driver: Fixed the restart loop of the CSI controller container, caused by dangling snapshots and volumes of a removed cluster. A gRPC keepalive and a connection timeout now detect a broken connection to a control plane instead of blocking a request.
Operator and CSI Driver: Updated the container base images and the Go dependencies to resolve open CVEs.

Fixes in 26.3.1

Operator: Fixed the deletion of a StoragePool with DHCHAP enabled, which stayed in Terminating because the update dropping its finalizer was rejected by its own validation.
Operator: Fixed the creation of a StoragePool with DHCHAP enabled and an empty allowedNodes, which is rejected now, since such a pool can never provision a volume.
Operator: Fixed the generated StorageClass of a DHCHAP pool, whose allowedTopologies term matched no node.
Operator: Fixed the label marking the allowed nodes of a StoragePool, which is keyed by the pool UUID now and can no longer exceed the 63 bytes a label name allows.
Operator: Fixed the failback of a replicated volume, whose target cluster, pool, and volume were not resolved through the relationship lookup.
Operator: Fixed the list of owned custom resource definitions in the cluster service version.
Operator: Updated golang.org/x/crypto and the etcd client to resolve three high severity advisories.
CSI Driver: Fixed the connection monitor, whose repair pass tore down a path that its reconcile pass had created moments earlier.

Important Changes

The storage plane sizing of a node moves to the cluster. The flags --max-subsys, --hugepages-mem, and --vcpu-count are set on sbctl cluster create and sbctl cluster add, and a node adopts them when it is added and on every restart. --max-subsys, --max-size, and --cores-percentage are removed from sbctl storage-node configure, and --max-subsys and --max-size from sbctl storage-node restart. The flag --max-lvol is renamed to --max-subsys, which is what it has always limited.
The core budget of the storage plane is an absolute vCPU count instead of a percentage. A node that cannot satisfy the count of its cluster, plus one core for the system, is refused instead of running on fewer cores than requested. sbctl cluster update --vcpu-count does not exist, because rewriting the core mask of every node belongs to a deliberate redeployment.
A configured subsystem limit above 75 per storage node is rejected, where it was accepted and then clamped silently before, so a node reserved huge pages for subsystems it could never serve. An lvstore serves at most 6000 objects, counting volumes, clones, and snapshots, and a create beyond that limit is refused.
Failure domains are a deploy-time feature. A cluster cannot be upgraded into them, and --enable-failure-domain has to be set at creation time. Once it is set, --failure-domain is mandatory on every sbctl storage-node add-node and rejected on a cluster without the feature.
sbctl storage-node remove no longer migrates the data of the node. The node has to host no volumes and no snapshots, and those are migrated beforehand. The flag --force-remove no longer removes volumes and snapshots, it cancels the active tasks of the node before the removal starts.
The commands sbctl storage-node resume, sbctl storage-node get-bdevs, and sbctl storage-node unload are removed. Instead of resume, sbctl storage-node restart has to be used.
The command sbctl cluster add-replication is deprecated. Instead, the commands sbctl cluster replication-target-add and sbctl cluster replication-policy-add have to be used, which support more than one destination per cluster.
sbctl volume migrate only pre-creates the target subsystem and returns the migration id and the NVMe connect strings. The flags --deadline and --max-retries move to sbctl volume migrate-continue, which advances the migration once the client is connected.
After sbctl cluster update, two commands have to be run. sbctl cluster upgrade-complete releases the journal compression hold the update took across its rolling restart and records the installed release, and sbctl cluster switch-write-protection moves the cluster to the second generation of distrib write protection, on which it does not land by itself.
The flag --all is removed from sbctl snapshot list, which gained --lvol-id, --node-id, and --json. The flag --cluster-id is removed from sbctl backup restore, whose cluster is derived from the backup.
The Pool resource is renamed to StoragePool, and the resource enrolling a group of workers is renamed from StorageNode to StorageNodeSet. The name StorageNode is now used for the per-worker resource that is created by the operator.
The field maxLogicalVolumeCount is renamed to maxSubsystemCount, and the field corePercentage is removed. Storage node sizing is moved from the StorageNodeSet to the cluster scope, and the field partitions is replaced by enableJournalDevice.
The deprecated StorageCluster fields blockSize, qpairCount, pageSizeInBlocks, maxQueueSize, inflightIOThreshold, strictNodeAntiAffinity, haType, isSingleNode, and clientQpairCount are removed.
The StorageCluster spec field maxFaultTolerance is replaced by maxConcurrentWorkerRestarts, which is capped at the fault tolerance of the cluster. The backend-reported maxFaultTolerance remains part of the status.
The field unpinBeforeDrain is removed. A pinned volume has to be unpinned manually before its storage node is drained.
The volume placement annotations are unified into simplyblock.io/selected-storage-node. On a new PVC, a host id supplied through simplyblock.io/host-id or simplybk/host-id is rewritten into that annotation and treated as a pin. On a pre-existing PVC, both legacy annotations are still honored as the lowest-priority source of the primary storage node.

Upgrade Considerations

The CSI driver added an extended check to prevent an accidental filesystem format onto an existing pre-formatted volume in one of the upstream CSI libraries. To ensure all existing volumes work as expected, users are highly encouraged to run this script after upgrading to 26.3.0. Annotate existing PVCs Script ⧉
An upgrade from the previous R25 Helm chart to the R26 Operator installation requires a maintenance window and additional steps. See the Upgrade Procedure R25 to R26 for details.

Known Issues

At the moment it is only possible to define one erasure coding schema per cluster.
When deleting pods with simplyblock-backed PVCs, it is important to delete the pod first, before deleting the PVC.
Deleting a pod before its persistent volume claim
```
kubectl delete pod <pod-name> --wait=true
kubectl delete pvc <pvc-name> --wait=true
```
A full FTT 1 or FTT 2 network outage can create a situation where surviving nodes lose access to the last journal records. This can cause loss of I/O availability until at least one node is restarted.
Node Removal currently only works with online nodes.

Features to Expect with Next Major Release

The ability to use different erasure coding schemas within the same cluster.
Inline data integrity validation to prevent silent data corruption.