Skip to content

26.4

Simplyblock is happy to release the third beta of Simplyblock 26.4. The changes are grouped by the component they belong to: the simplyblock control plane and storage plane, and the Kubernetes components, meaning the Simplyblock Operator and Simplyblock CSI. Within each component, the changes of 26.4.0-beta1 are listed first, followed by those of 26.4.0-beta2 and 26.4.0-beta3.

Beta Releases

This page is a work in progress. The upgrade path and the known issues are not final yet.

Simplyblock

The control plane and the storage plane are released together and share a version number. They are deployed and operated by the Simplyblock Operator, so the changes below reach a cluster with the operator release of the same version.

New Features in 26.4.0-beta1

  • Control Plane: Added support for Linux block devices, such as SAS or SATA SSDs, partitions, and cloud volumes, as storage instead of NVMe PCIe devices (Linux Block Devices).
  • Control Plane: Added support for single-node clusters, which run without high availability.
  • Storage Plane: Added optional inline checksum validation, by which silent data corruption is detected.
  • Control Plane: Added consistency groups, whose volumes are snapshotted, replicated, and failed over together.
  • Control Plane: Added tiered snapshot retention and a declared recovery point objective to replication policies, and Prometheus metrics for replication.
  • Control Plane: Added secondary indices to the FoundationDB store of the control plane, so that lookups no longer slow down as a cluster grows. The indices are built by the operator when the database starts.
  • Control Plane: Added several alerting contact points at once, such as Slack, Microsoft Teams, PagerDuty, Opsgenie, and a webhook.
  • Control Plane: Added change streaming to the v2 management API, and v2 endpoints to add, replace, and fail a device.
  • Storage Plane: The NVMe-oF target allows 64 I/O queue pairs per controller instead of 32.

Fixes in 26.4.0-beta1

  • Control Plane: Fixed several failover and failback issues of replicated volumes and consistency groups, including the loss of the last recovery point when a volume was detached.
  • Control Plane: Improved the reliability of the batch migration and the cutover of namespaced volumes.
  • Control Plane: Fixed a cluster without parity that reported degraded while healthy, and the huge page reservation of a new storage node, which was twice the required size.
  • Control Plane: Fixed DH-CHAP keys and other secrets leaking into logs and API responses.
  • Control Plane: Fixed the Kubernetes node preparation on Ubuntu, whose kernel module job never succeeded.
  • General: Updated oauthlib and urllib3 to resolve open CVEs.

New Features in 26.4.0-beta2

  • Control Plane: Added arbitration for two-node clusters, by which an arbiter decides which node keeps serving after a failure of the other (Edge Clusters).
  • Control Plane: Added backups taken directly from a volume, without a snapshot having to be created first. The same is available through the v2 management API.
  • Control Plane: Improved the replication, which is sent from any online member of the source now.
  • Storage Plane: The lvstore poller group runs on the NVMe-oF poller cores instead of a single core.

Fixes in 26.4.0-beta2

  • Control Plane: Fixed a stalled S3 backup transfer, which is retried now instead of aborted, and the chaining of automatic backups.
  • Control Plane: Fixed an ANA failover that was skipped and never retried, and a migration that was retried on a cluster that could not operate.
  • Control Plane: Fixed the detection of the partition table when the journal of a Linux block device node is split.
  • General: Updated the Go toolchain of the event collector and a transitive dependency to resolve open CVEs.

New Features in 26.4.0-beta3

  • Control Plane: Added host fencing for two-node clusters. When the arbiter grants a node, the NVMe host identities of the other node's Kubernetes worker are revoked on the granted node, so that its workloads can be restarted without a second writer (Edge Clusters).
  • Control Plane: Added a service class reserved for replication traffic, with a configurable share of the cluster's I/O, and the declarative configuration of a cluster's service classes through the v2 management API (Performance and QoS).
  • Control Plane: Replication sends only the changes since the last replicated snapshot, which are merged on the target. Consecutive replication cycles overlap.
  • Control Plane: Added the online migration of a cluster's volumes to another cluster, with migration plans, a forecast of the convergence, and a live cutover.
  • Control Plane: Added backups of consistency groups, which back up one generation of a group as a single unit and restore it into a new group. Group backups can be exported and imported across clusters, and a backup policy can be attached to a group.
  • Control Plane: Added the forced deletion of a cluster, which shuts down and removes its storage nodes and pools first.

Fixes in 26.4.0-beta3

  • Control Plane: Fixed the used size of volumes on an lvstore created by 26.3.1 or earlier, which was reported up to twice the real size.
  • Control Plane: Fixed the creation of volumes on a single-node cluster, and the addition of storage nodes to a second cluster on the same control plane.
  • Control Plane: Fixed several fail-back issues of replicated volumes, including the deletion of a replicated copy that a failed-over volume was cloned from.
  • Control Plane: Fixed a failed migration that was not rolled back, and a storage node restart that waited on a stopped SPDK pod.
  • Storage Plane: Fixed an I/O hang when many large I/Os were split at once, the handling of a resubmitted unmap request, and the QoS scheduler of the device layer.
  • General: Updated certifi and the Go toolchain to resolve open CVEs.

Kubernetes

The changes below apply to a Kubernetes-based deployment and ship with the Simplyblock Operator and Simplyblock CSI.

New Features in 26.4.0-beta1

  • Operator: The CSI driver is deployed and configured through the new SimplyblockDriver resource.
  • Operator: Added support for Linux block devices as storage of a storage node (Linux Block Devices).
  • Operator: Added inline checksum validation, enabled with enableChecksumValidation on the StorageCluster.
  • CSI Driver: Added client-side compression and deduplication with VDO, enabled with enableClientCompression and enableClientDeduplication on the StoragePool (Storage Classes).
  • CSI Driver: Added crash-consistent group snapshots through the Kubernetes VolumeGroupSnapshot, and the restore of a whole group with VolumeGroupSnapshotOps (Snapshotting).
  • Operator: Added disaster recovery across clusters through csi-addons volume replication and Ramen, operated from the new Control Center (Disaster Recovery).
  • Operator: Added alerting on the cluster event stream and additional notification contact points.
  • Operator: Improved the latency baseline of the volume rebalancer, which now rejects outliers.
  • Operator: The minimum vcpuCount of a storage node is lowered from 6 to 4.

Fixes in 26.4.0-beta1

  • Operator: Improved the storage node drain and removal, which retry a failed move on another node, respect failure domains, and wait for the removal to be admitted.
  • Operator: Fixed the rebalancing after a cluster expansion, and a StorageBackupOps restore that was retried indefinitely.
  • Operator: Fixed the logs of operator-deployed workloads, which no longer reached Graylog, and the capacity alerts, which lost their cluster label.
  • CSI Driver: Fixed a hung device blocking the node plugin, and improved the connecting and disconnecting of multipath volumes.
  • Operator and CSI Driver: Updated gRPC, golang.org/x/crypto, and the etcd client to resolve open CVEs.

New Features in 26.4.0-beta2

  • CSI Driver: Added ReadWriteMany volumes, served through pNFS (ReadWriteMany Volumes).
  • Operator: Added edge clusters of one or two storage nodes, with arbitration for a two-node cluster enabled by enableArbitration (Deploying Edge Clusters).
  • Operator: Added on-demand backups, requested with a StorageBackup that names a claim, and restores across namespaces (Backup and Recovery).
  • Operator: Added AI-assisted discovery and disaster recovery proposals to the Control Center (Using Discovery).
  • Operator: In-cluster addresses are built as <service>.<namespace>.svc, which resolves on any cluster DNS domain. The domain needs no configuration, and is read from the pod's resolver where a name must be fully qualified.

Fixes in 26.4.0-beta2

  • Operator: Fixed a maintenance window that did not wait for the worker to be uncordoned, and restarted a storage node agent that stops answering.
  • Operator: Fixed the Helm chart, which now renders a StorageClass for every storage cluster created at installation and releases volumes before the CSI driver is uninstalled.
  • CSI Driver: Fixed the recovery of a volume whose staging was interrupted.
  • Operator: Fixed the published Helm repository, from which earlier chart releases were missing.
  • Operator and CSI Driver: Updated golang.org/x/net to resolve open CVEs.

New Features in 26.4.0-beta3

  • Operator: Added superclusters, whose members are storage clusters on the same control plane. Volumes are placed among the members, and a volume is moved to another member with the PersistentVolumeOps action Relocate (Superclusters).
  • Operator: Added host fencing to two-node edge clusters, enabled with arbitration.hostFencing. The workloads of a fenced node are restarted on the other node through a NodeHealthCheck and the new StorageFenceRemediation (Deploying Edge Clusters).
  • Operator: Added storage networks attached through Multus NetworkAttachmentDefinitions, which the discovery run detects and writes into the ClusterDeploymentConfig.
  • Operator: Added support for Amazon EKS (Amazon EKS).
  • Operator: The CSI link uses the TLS settings of the installation, including mutual TLS.

Fixes in 26.4.0-beta3

  • Operator: Fixed a volume migration that stalled when it was aborted late, and migration jobs that ran a rebalancer of another release.
  • Operator: Fixed a failure domain that was sent to a cluster without the feature, and the order in which an expansion adds nodes across failure domains.
  • CSI Driver: Fixed the repair of NVMe paths, which tore down a controller that had just connected.
  • Operator: Fixed the Helm chart, of which several values were not read by any template.

Important Changes

Simplyblock 26.4 is released for Kubernetes-based deployments only.

The operator API moves from storage.simplyblock.io/v1alpha1 to storage.simplyblock.io/v1alpha2. Every kind is served at the new version, which is also the storage version, and a conversion webhook translates an existing v1alpha1 object so that a manifest written for 26.3 keeps working. The replication kinds are the exception and stay at v1alpha1.

  • Imperative operations are separate resources instead of fields on the entity they act on. Setting StorageCluster.spec.action is replaced by creating a StorageClusterOps, and the operation runs to a terminal phase and stays afterward as the record of what was done. StorageClusterOps, StorageNodeOps, StorageDeviceOps, StoragePoolOps, PersistentVolumeOps, StorageBackupOps, ControlPlaneOps, and OperatorOps share that shape, including spec.abort and the Aborted phase.
  • A storage cluster is deployed from a ClusterDeploymentConfig. The operator inspects the workers, writes the draft, and an administrator approves it with spec.approved, after which the document is expanded into the StorageCluster and its StorageNode objects. The document is immutable once approved.
  • A namespace holds at most one StorageCluster. A second storage cluster needs a namespace of its own.
  • Every enum value of the group is PascalCase. An action such as restart is written Restart, and a phase such as Completed is written Succeeded.
  • The status of a running operation is a state machine snapshot in status.step, replacing status.subPhase. status.step.state carries the step, and status.step.deadline is when that step expires.
  • The annotation and label keys move from the simplyblock.io/ prefix to storage.simplyblock.io/. Every spelling is read and the new one is written, so nothing has to be relabeled. The exception is simplyblock.io/pod-affinity, which has no prefixed spelling yet.
  • The StorageCluster finalizer is renamed from storage.simplyblock.io/cluster-finalizer to storage.simplyblock.io/storagecluster-finalizer. The old key is read for one release.

Renamed and Retired Resources

A kind that is absorbed is not converted. The upgrade constructs the successor, and a manifest that creates the old kind has to be rewritten.

26.3 resource 26.4 resource Notes
VolumeMigration PersistentVolumeOps Cluster-scoped, because a PersistentVolume is. Action Migrate.
BackupRestore StorageBackupOps Absorbed as spec.action: Restore, the kind's only action.
BackupPolicy StorageBackupPolicy Selects its claims by label instead of a per-claim annotation.
BackupImport Retired The backup store is the inventory, so a backup another cluster wrote needs no import.
StorageNodeSet StorageCluster.spec.storageNodes The fleet template. The workers move to ClusterDeploymentConfig.
Task StorageCluster.status.tasks Canceled with a StorageClusterOps carrying action: CancelTask.

StorageCluster

26.3 property 26.4 property Notes
spec.maxHugePagesSize spec.minHugePagesSize The value is a minimum and always was.
spec.hashicorpVaultSettings.baseURL spec.kms.vault.endpoint
spec.backup.localEndpoint spec.backup.endpoint bucket, prefix, and region are added.
spec.openShiftCluster spec.storageNodes.openshift
spec.storageNodes.openShiftMachineConfigPool spec.storageNodes.openshift.machineConfigPool
spec.enableDataRealignment spec.disableDataRealignment Inverted. Realignment is on by default now.
spec.volumeAutoPlacement.enabled spec.enableVolumeAutoPlacement Moves up to the cluster spec.
spec.volumeAutoPlacement.migrationEnabled spec.volumeAutoPlacement.disableMigration Inverted.
spec.volumeAutoPlacement.latencyBenchmarkEnabled spec.volumeAutoPlacement.enableLatencyBenchmark
spec.volumeMigrationSettings.enabled Removed Migration cannot be turned off.
spec.action StorageClusterOps
status.actionStatus StorageClusterOps.status

An inverted field is the dangerous row of this release. A spec that still carries enableDataRealignment or migrationEnabled is accepted by the API server and ignored, and the feature runs with the opposite default.

StorageClusterOps

26.3 property 26.4 property
spec.nodeRollingRestart spec.rollingRestart
status.nodeRollingRestartStatus status.rollingRestart
status.triggered Removed

The action node-recycle is renamed to RollingRestart, and activate, expand, shutdown, start, and restart become Activate, Expand, Shutdown, Start, and Restart. CancelTask is added.

StorageNode

26.3 property 26.4 property Notes
spec.storageNodeSetRef spec.clusterRef and spec.nodeSet The node is owned by its cluster.
spec.socketIndex spec.slot
spec.overrides spec.config
config.failureDomain config.failureDomain Was an integer index, is a label value such as rack-a.
spec.skipKubeletConfiguration StorageCluster.spec.storageNodes.enableKubeletConfiguration Inverted, and per cluster.
overrides.enableCpuTopology StorageCluster.spec.storageNodes.enableCpuTopology Per cluster, not per node.

StorageNodeOps

26.3 property 26.4 property Notes
spec.storageNodeRef spec.nodeRef
spec.targetWorkerNode spec.migrate.targetWorkerNode
spec.newSsdPcie spec.migrate.newSsdPcie
spec.drain spec.remove Matches the action it parameterizes.
status.subPhase status.step.state
status.volumesMigrated status.drain.volumesMigrated
status.volumesPending status.drain.volumesTotal A total instead of a remainder.
status.triggered Removed

The six actions become Shutdown, Restart, Suspend, Resume, Remove, and Migrate, and HostMaintenance is added. The sub-phases Migrating and Restarting of a migration become the steps Relocating and AwaitingNode, and the Migrating sub-phase of a removal becomes MigratingVolumes.

StoragePool

26.3 property 26.4 property
spec.clusterName spec.clusterRef
spec.capacityLimit spec.limits.capacity
spec.logicalVolumeMaxSize spec.limits.maxVolumeSize
spec.qos.iops spec.limits.iops
spec.qos.throughput spec.limits.throughput
spec.dhchap spec.volumeDefaults.enableDHCHAP
spec.storageClassParameters.qosRwIops spec.volumeDefaults.iops
spec.storageClassParameters.qosRwMbytes spec.volumeDefaults.throughput.readWrite
spec.storageClassParameters.qosRMbytes spec.volumeDefaults.throughput.read
spec.storageClassParameters.qosWMbytes spec.volumeDefaults.throughput.write
spec.storageClassParameters.encryption spec.volumeDefaults.enableEncryption
spec.storageClassParameters.maxNamespacePerSubsys spec.volumeDefaults.maxNamespacesPerSubsystem
spec.storageClassParameters.fabric spec.volumeDefaults.fabric
spec.storageClassParameters.filesystem spec.volumeDefaults.filesystem
spec.storageClassParameters.tune2fsReservedBlocks spec.volumeDefaults.tune2fsReservedBlocks
spec.action, spec.status Removed

spec.limits is mutable, so a capacity or a QoS ceiling is raised on a running pool. spec.volumeDefaults is immutable, because the parameters of the generated StorageClass are.

StorageBackup, StorageBackupPolicy, and PersistentVolumeOps

26.3 property 26.4 property Notes
StorageBackup.spec.clusterName spec.clusterRef
StorageBackup.spec.pvcRef spec.source.claimName Requests a backup. Other objects are discovered.
StorageBackup flat status fields status.backup and status.source The copy and the volume it was taken from.
BackupPolicy.spec.clusterName StorageBackupPolicy.spec.clusterRef
BackupPolicy.status.attachedLvols StorageBackupPolicy.status.attachedClaims
VolumeMigration.spec.pvName PersistentVolumeOps.spec.persistentVolumeName
VolumeMigration.spec.targetNodeUUID PersistentVolumeOps.spec.migrate.targetNodeRef Names a StorageNode object, not a UUID.
VolumeMigration phase Completed PersistentVolumeOps phase Succeeded

A 26.3 StorageBackup becomes a request only when its pvcRef names a claim in its own namespace and no backup ID has been recorded yet. Every other object becomes a record, and a record the backup store does not report is removed.

ControlPlane

26.3 property 26.4 property Notes
spec.image spec.source.local.image spec.source.managed names a remote control plane instead.
phase Ready phase Available Degraded and Unavailable are added.

StorageClass Parameters and Claim Annotations

The QoS keys are renamed outside the CRD schemas as well. Every generation is read, because the parameters of a StorageClass cannot be changed after it is created, so a class an older release generated keeps working.

26.3 parameter 26.4 parameter 26.4 claim annotation
qos_rw_iops max_iops storage.simplyblock.io/max-iops
qos_rw_mbytes max_mbytes_per_sec storage.simplyblock.io/max-mbytes-per-sec
qos_r_mbytes max_read_mbytes_per_sec storage.simplyblock.io/max-read-mbytes-per-sec
qos_w_mbytes max_write_mbytes_per_sec storage.simplyblock.io/max-write-mbytes-per-sec

A StorageClass is assigned to a storage pool by three labels, storage.simplyblock.io/namespace, storage.simplyblock.io/cluster, and storage.simplyblock.io/pool, instead of being named after the pool. A pool may have any number of classes. The parameter dhchap_node_label is renamed to dhchap_node_selector, and the node label it names changes from simplyblock.io/pool.<namespace>.<cluster>.<pool> to storage.simplyblock.io/storage-pool.<pool-uuid>.

Other Changes in 26.4.0-beta1

  • The CSI driver no longer passes on the StorageClass parameters compression, replicate, and priority_class.
  • The default filesystem of a volume is xfs, and the image pull policy of the storage nodes and the CSI driver defaults to Always.
  • The StorageCluster fields spec.storageNodes.dataInterfaces and spec.enableNodeAffinity cannot be changed once set.

Changes in 26.4.0-beta2

  • The metrics endpoint of the storage node proxy no longer requires authentication.
  • The csi-addons sidecar is deployed only when enableCSIAddons is set on the SimplyblockDriver. A deployment that uses csi-addons volume replication has to set it.
  • The bundled csi-driver-host-path and its local-hostpath StorageClass are removed from the Helm chart, along with controlplane.csiHostpathDriver.enabled.

Changes in 26.4.0-beta3

  • The maximum number of NVMe-oF subsystems per storage node is raised from 75 to 500 (Limits). Each subsystem reserves huge page memory, so maxSubsystemCount should reflect the planned number of volumes.
  • The TLS settings of the installation are read from ControlPlane.spec.source.local.tls, which cannot be changed once set. The environment variables SB_TLS_SERVE, SB_TLS_CONNECT, SB_TLS_CLIENT_AUTH, and FDB_TLS_* are removed from the operator.
  • The initial discovery run is an OperatorOps with the action Discover, and the Helm values discovery.* are the fields of its spec.discover. The simplyblock-bootstrap ConfigMap is no longer created.
  • The control plane workloads and the CSI controller run on the pod network instead of the host network.
  • enableArbitration is passed to the control plane. A two-node cluster that set it with an earlier release runs with arbitration after the upgrade.
  • Container images are accepted from any registry.

Upgrade Considerations

  • TODO, to be confirmed once the simplyblock-upgrade tool is released.

Known Issues

  • TODO, to be re-checked against the 26.3 known issues.

Features to Expect with Next Major Release

  • The ability to use different erasure coding schemas within the same cluster.