26.4
Simplyblock is happy to release the third beta of Simplyblock 26.4. The changes are grouped by the component they belong to: the simplyblock control plane and storage plane, and the Kubernetes components, meaning the Simplyblock Operator and Simplyblock CSI. Within each component, the changes of 26.4.0-beta1 are listed first, followed by those of 26.4.0-beta2 and 26.4.0-beta3.
Beta Releases
This page is a work in progress. The upgrade path and the known issues are not final yet.
Simplyblock
The control plane and the storage plane are released together and share a version number. They are deployed and operated by the Simplyblock Operator, so the changes below reach a cluster with the operator release of the same version.
New Features in 26.4.0-beta1
- Control Plane: Added support for Linux block devices, such as SAS or SATA SSDs, partitions, and cloud volumes, as storage instead of NVMe PCIe devices (Linux Block Devices).
- Control Plane: Added support for single-node clusters, which run without high availability.
- Storage Plane: Added optional inline checksum validation, by which silent data corruption is detected.
- Control Plane: Added consistency groups, whose volumes are snapshotted, replicated, and failed over together.
- Control Plane: Added tiered snapshot retention and a declared recovery point objective to replication policies, and Prometheus metrics for replication.
- Control Plane: Added secondary indices to the FoundationDB store of the control plane, so that lookups no longer slow down as a cluster grows. The indices are built by the operator when the database starts.
- Control Plane: Added several alerting contact points at once, such as Slack, Microsoft Teams, PagerDuty, Opsgenie, and a webhook.
- Control Plane: Added change streaming to the v2 management API, and v2 endpoints to add, replace, and fail a device.
- Storage Plane: The NVMe-oF target allows 64 I/O queue pairs per controller instead of 32.
Fixes in 26.4.0-beta1
- Control Plane: Fixed several failover and failback issues of replicated volumes and consistency groups, including the loss of the last recovery point when a volume was detached.
- Control Plane: Improved the reliability of the batch migration and the cutover of namespaced volumes.
- Control Plane: Fixed a cluster without parity that reported
degradedwhile healthy, and the huge page reservation of a new storage node, which was twice the required size. - Control Plane: Fixed DH-CHAP keys and other secrets leaking into logs and API responses.
- Control Plane: Fixed the Kubernetes node preparation on Ubuntu, whose kernel module job never succeeded.
- General: Updated
oauthlibandurllib3to resolve open CVEs.
New Features in 26.4.0-beta2
- Control Plane: Added arbitration for two-node clusters, by which an arbiter decides which node keeps serving after a failure of the other (Edge Clusters).
- Control Plane: Added backups taken directly from a volume, without a snapshot having to be created first. The same is available through the v2 management API.
- Control Plane: Improved the replication, which is sent from any online member of the source now.
- Storage Plane: The lvstore poller group runs on the NVMe-oF poller cores instead of a single core.
Fixes in 26.4.0-beta2
- Control Plane: Fixed a stalled S3 backup transfer, which is retried now instead of aborted, and the chaining of automatic backups.
- Control Plane: Fixed an ANA failover that was skipped and never retried, and a migration that was retried on a cluster that could not operate.
- Control Plane: Fixed the detection of the partition table when the journal of a Linux block device node is split.
- General: Updated the Go toolchain of the event collector and a transitive dependency to resolve open CVEs.
New Features in 26.4.0-beta3
- Control Plane: Added host fencing for two-node clusters. When the arbiter grants a node, the NVMe host identities of the other node's Kubernetes worker are revoked on the granted node, so that its workloads can be restarted without a second writer (Edge Clusters).
- Control Plane: Added a service class reserved for replication traffic, with a configurable share of the cluster's I/O, and the declarative configuration of a cluster's service classes through the v2 management API (Performance and QoS).
- Control Plane: Replication sends only the changes since the last replicated snapshot, which are merged on the target. Consecutive replication cycles overlap.
- Control Plane: Added the online migration of a cluster's volumes to another cluster, with migration plans, a forecast of the convergence, and a live cutover.
- Control Plane: Added backups of consistency groups, which back up one generation of a group as a single unit and restore it into a new group. Group backups can be exported and imported across clusters, and a backup policy can be attached to a group.
- Control Plane: Added the forced deletion of a cluster, which shuts down and removes its storage nodes and pools first.
Fixes in 26.4.0-beta3
- Control Plane: Fixed the used size of volumes on an lvstore created by 26.3.1 or earlier, which was reported up to twice the real size.
- Control Plane: Fixed the creation of volumes on a single-node cluster, and the addition of storage nodes to a second cluster on the same control plane.
- Control Plane: Fixed several fail-back issues of replicated volumes, including the deletion of a replicated copy that a failed-over volume was cloned from.
- Control Plane: Fixed a failed migration that was not rolled back, and a storage node restart that waited on a stopped SPDK pod.
- Storage Plane: Fixed an I/O hang when many large I/Os were split at once, the handling of a resubmitted unmap request, and the QoS scheduler of the device layer.
- General: Updated
certifiand the Go toolchain to resolve open CVEs.
Kubernetes
The changes below apply to a Kubernetes-based deployment and ship with the Simplyblock Operator and Simplyblock CSI.
New Features in 26.4.0-beta1
- Operator: The CSI driver is deployed and configured through the new
SimplyblockDriverresource. - Operator: Added support for Linux block devices as storage of a storage node (Linux Block Devices).
- Operator: Added inline checksum validation, enabled with
enableChecksumValidationon theStorageCluster. - CSI Driver: Added client-side compression and deduplication with VDO, enabled with
enableClientCompressionandenableClientDeduplicationon theStoragePool(Storage Classes). - CSI Driver: Added crash-consistent group snapshots through the Kubernetes
VolumeGroupSnapshot, and the restore of a whole group withVolumeGroupSnapshotOps(Snapshotting). - Operator: Added disaster recovery across clusters through csi-addons volume replication and Ramen, operated from the new Control Center (Disaster Recovery).
- Operator: Added alerting on the cluster event stream and additional notification contact points.
- Operator: Improved the latency baseline of the volume rebalancer, which now rejects outliers.
- Operator: The minimum
vcpuCountof a storage node is lowered from 6 to 4.
Fixes in 26.4.0-beta1
- Operator: Improved the storage node drain and removal, which retry a failed move on another node, respect failure domains, and wait for the removal to be admitted.
- Operator: Fixed the rebalancing after a cluster expansion, and a
StorageBackupOpsrestore that was retried indefinitely. - Operator: Fixed the logs of operator-deployed workloads, which no longer reached Graylog, and the capacity alerts, which lost their cluster label.
- CSI Driver: Fixed a hung device blocking the node plugin, and improved the connecting and disconnecting of multipath volumes.
- Operator and CSI Driver: Updated gRPC,
golang.org/x/crypto, and the etcd client to resolve open CVEs.
New Features in 26.4.0-beta2
- CSI Driver: Added
ReadWriteManyvolumes, served through pNFS (ReadWriteMany Volumes). - Operator: Added edge clusters of one or two storage nodes, with arbitration for a two-node cluster enabled by
enableArbitration(Deploying Edge Clusters). - Operator: Added on-demand backups, requested with a
StorageBackupthat names a claim, and restores across namespaces (Backup and Recovery). - Operator: Added AI-assisted discovery and disaster recovery proposals to the Control Center (Using Discovery).
- Operator: In-cluster addresses are built as
<service>.<namespace>.svc, which resolves on any cluster DNS domain. The domain needs no configuration, and is read from the pod's resolver where a name must be fully qualified.
Fixes in 26.4.0-beta2
- Operator: Fixed a maintenance window that did not wait for the worker to be uncordoned, and restarted a storage node agent that stops answering.
- Operator: Fixed the Helm chart, which now renders a
StorageClassfor every storage cluster created at installation and releases volumes before the CSI driver is uninstalled. - CSI Driver: Fixed the recovery of a volume whose staging was interrupted.
- Operator: Fixed the published Helm repository, from which earlier chart releases were missing.
- Operator and CSI Driver: Updated
golang.org/x/netto resolve open CVEs.
New Features in 26.4.0-beta3
- Operator: Added superclusters, whose members are storage clusters on the same control plane. Volumes are
placed among the members, and a volume is moved to another member with the
PersistentVolumeOpsactionRelocate(Superclusters). - Operator: Added host fencing to two-node edge clusters, enabled with
arbitration.hostFencing. The workloads of a fenced node are restarted on the other node through a NodeHealthCheck and the newStorageFenceRemediation(Deploying Edge Clusters). - Operator: Added storage networks attached through Multus
NetworkAttachmentDefinitions, which the discovery run detects and writes into theClusterDeploymentConfig. - Operator: Added support for Amazon EKS (Amazon EKS).
- Operator: The CSI link uses the TLS settings of the installation, including mutual TLS.
Fixes in 26.4.0-beta3
- Operator: Fixed a volume migration that stalled when it was aborted late, and migration jobs that ran a rebalancer of another release.
- Operator: Fixed a failure domain that was sent to a cluster without the feature, and the order in which an expansion adds nodes across failure domains.
- CSI Driver: Fixed the repair of NVMe paths, which tore down a controller that had just connected.
- Operator: Fixed the Helm chart, of which several values were not read by any template.
Important Changes
Simplyblock 26.4 is released for Kubernetes-based deployments only.
The operator API moves from storage.simplyblock.io/v1alpha1 to storage.simplyblock.io/v1alpha2. Every kind is
served at the new version, which is also the storage version, and a conversion webhook translates an existing
v1alpha1 object so that a manifest written for 26.3 keeps working. The replication kinds are the exception and
stay at v1alpha1.
- Imperative operations are separate resources instead of fields on the entity they act on. Setting
StorageCluster.spec.actionis replaced by creating aStorageClusterOps, and the operation runs to a terminal phase and stays afterward as the record of what was done.StorageClusterOps,StorageNodeOps,StorageDeviceOps,StoragePoolOps,PersistentVolumeOps,StorageBackupOps,ControlPlaneOps, andOperatorOpsshare that shape, includingspec.abortand theAbortedphase. - A storage cluster is deployed from a
ClusterDeploymentConfig. The operator inspects the workers, writes the draft, and an administrator approves it withspec.approved, after which the document is expanded into theStorageClusterand itsStorageNodeobjects. The document is immutable once approved. - A namespace holds at most one
StorageCluster. A second storage cluster needs a namespace of its own. - Every enum value of the group is PascalCase. An action such as
restartis writtenRestart, and a phase such asCompletedis writtenSucceeded. - The status of a running operation is a state machine snapshot in
status.step, replacingstatus.subPhase.status.step.statecarries the step, andstatus.step.deadlineis when that step expires. - The annotation and label keys move from the
simplyblock.io/prefix tostorage.simplyblock.io/. Every spelling is read and the new one is written, so nothing has to be relabeled. The exception issimplyblock.io/pod-affinity, which has no prefixed spelling yet. - The
StorageClusterfinalizer is renamed fromstorage.simplyblock.io/cluster-finalizertostorage.simplyblock.io/storagecluster-finalizer. The old key is read for one release.
Renamed and Retired Resources
A kind that is absorbed is not converted. The upgrade constructs the successor, and a manifest that creates the old kind has to be rewritten.
| 26.3 resource | 26.4 resource | Notes |
|---|---|---|
VolumeMigration |
PersistentVolumeOps |
Cluster-scoped, because a PersistentVolume is. Action Migrate. |
BackupRestore |
StorageBackupOps |
Absorbed as spec.action: Restore, the kind's only action. |
BackupPolicy |
StorageBackupPolicy |
Selects its claims by label instead of a per-claim annotation. |
BackupImport |
Retired | The backup store is the inventory, so a backup another cluster wrote needs no import. |
StorageNodeSet |
StorageCluster.spec.storageNodes |
The fleet template. The workers move to ClusterDeploymentConfig. |
Task |
StorageCluster.status.tasks |
Canceled with a StorageClusterOps carrying action: CancelTask. |
StorageCluster
| 26.3 property | 26.4 property | Notes |
|---|---|---|
spec.maxHugePagesSize |
spec.minHugePagesSize |
The value is a minimum and always was. |
spec.hashicorpVaultSettings.baseURL |
spec.kms.vault.endpoint |
|
spec.backup.localEndpoint |
spec.backup.endpoint |
bucket, prefix, and region are added. |
spec.openShiftCluster |
spec.storageNodes.openshift |
|
spec.storageNodes.openShiftMachineConfigPool |
spec.storageNodes.openshift.machineConfigPool |
|
spec.enableDataRealignment |
spec.disableDataRealignment |
Inverted. Realignment is on by default now. |
spec.volumeAutoPlacement.enabled |
spec.enableVolumeAutoPlacement |
Moves up to the cluster spec. |
spec.volumeAutoPlacement.migrationEnabled |
spec.volumeAutoPlacement.disableMigration |
Inverted. |
spec.volumeAutoPlacement.latencyBenchmarkEnabled |
spec.volumeAutoPlacement.enableLatencyBenchmark |
|
spec.volumeMigrationSettings.enabled |
Removed | Migration cannot be turned off. |
spec.action |
StorageClusterOps |
|
status.actionStatus |
StorageClusterOps.status |
An inverted field is the dangerous row of this release. A spec that still carries enableDataRealignment or
migrationEnabled is accepted by the API server and ignored, and the feature runs with the opposite default.
StorageClusterOps
| 26.3 property | 26.4 property |
|---|---|
spec.nodeRollingRestart |
spec.rollingRestart |
status.nodeRollingRestartStatus |
status.rollingRestart |
status.triggered |
Removed |
The action node-recycle is renamed to RollingRestart, and activate, expand, shutdown, start, and
restart become Activate, Expand, Shutdown, Start, and Restart. CancelTask is added.
StorageNode
| 26.3 property | 26.4 property | Notes |
|---|---|---|
spec.storageNodeSetRef |
spec.clusterRef and spec.nodeSet |
The node is owned by its cluster. |
spec.socketIndex |
spec.slot |
|
spec.overrides |
spec.config |
|
config.failureDomain |
config.failureDomain |
Was an integer index, is a label value such as rack-a. |
spec.skipKubeletConfiguration |
StorageCluster.spec.storageNodes.enableKubeletConfiguration |
Inverted, and per cluster. |
overrides.enableCpuTopology |
StorageCluster.spec.storageNodes.enableCpuTopology |
Per cluster, not per node. |
StorageNodeOps
| 26.3 property | 26.4 property | Notes |
|---|---|---|
spec.storageNodeRef |
spec.nodeRef |
|
spec.targetWorkerNode |
spec.migrate.targetWorkerNode |
|
spec.newSsdPcie |
spec.migrate.newSsdPcie |
|
spec.drain |
spec.remove |
Matches the action it parameterizes. |
status.subPhase |
status.step.state |
|
status.volumesMigrated |
status.drain.volumesMigrated |
|
status.volumesPending |
status.drain.volumesTotal |
A total instead of a remainder. |
status.triggered |
Removed |
The six actions become Shutdown, Restart, Suspend, Resume, Remove, and Migrate, and
HostMaintenance is added. The sub-phases Migrating and Restarting of a migration become the steps
Relocating and AwaitingNode, and the Migrating sub-phase of a removal becomes MigratingVolumes.
StoragePool
| 26.3 property | 26.4 property |
|---|---|
spec.clusterName |
spec.clusterRef |
spec.capacityLimit |
spec.limits.capacity |
spec.logicalVolumeMaxSize |
spec.limits.maxVolumeSize |
spec.qos.iops |
spec.limits.iops |
spec.qos.throughput |
spec.limits.throughput |
spec.dhchap |
spec.volumeDefaults.enableDHCHAP |
spec.storageClassParameters.qosRwIops |
spec.volumeDefaults.iops |
spec.storageClassParameters.qosRwMbytes |
spec.volumeDefaults.throughput.readWrite |
spec.storageClassParameters.qosRMbytes |
spec.volumeDefaults.throughput.read |
spec.storageClassParameters.qosWMbytes |
spec.volumeDefaults.throughput.write |
spec.storageClassParameters.encryption |
spec.volumeDefaults.enableEncryption |
spec.storageClassParameters.maxNamespacePerSubsys |
spec.volumeDefaults.maxNamespacesPerSubsystem |
spec.storageClassParameters.fabric |
spec.volumeDefaults.fabric |
spec.storageClassParameters.filesystem |
spec.volumeDefaults.filesystem |
spec.storageClassParameters.tune2fsReservedBlocks |
spec.volumeDefaults.tune2fsReservedBlocks |
spec.action, spec.status |
Removed |
spec.limits is mutable, so a capacity or a QoS ceiling is raised on a running pool. spec.volumeDefaults is
immutable, because the parameters of the generated StorageClass are.
StorageBackup, StorageBackupPolicy, and PersistentVolumeOps
| 26.3 property | 26.4 property | Notes |
|---|---|---|
StorageBackup.spec.clusterName |
spec.clusterRef |
|
StorageBackup.spec.pvcRef |
spec.source.claimName |
Requests a backup. Other objects are discovered. |
StorageBackup flat status fields |
status.backup and status.source |
The copy and the volume it was taken from. |
BackupPolicy.spec.clusterName |
StorageBackupPolicy.spec.clusterRef |
|
BackupPolicy.status.attachedLvols |
StorageBackupPolicy.status.attachedClaims |
|
VolumeMigration.spec.pvName |
PersistentVolumeOps.spec.persistentVolumeName |
|
VolumeMigration.spec.targetNodeUUID |
PersistentVolumeOps.spec.migrate.targetNodeRef |
Names a StorageNode object, not a UUID. |
VolumeMigration phase Completed |
PersistentVolumeOps phase Succeeded |
A 26.3 StorageBackup becomes a request only when its pvcRef names a claim in its own namespace and no backup ID
has been recorded yet. Every other object becomes a record, and a record the backup store does not report is
removed.
ControlPlane
| 26.3 property | 26.4 property | Notes |
|---|---|---|
spec.image |
spec.source.local.image |
spec.source.managed names a remote control plane instead. |
phase Ready |
phase Available |
Degraded and Unavailable are added. |
StorageClass Parameters and Claim Annotations
The QoS keys are renamed outside the CRD schemas as well. Every generation is read, because the parameters of a
StorageClass cannot be changed after it is created, so a class an older release generated keeps working.
| 26.3 parameter | 26.4 parameter | 26.4 claim annotation |
|---|---|---|
qos_rw_iops |
max_iops |
storage.simplyblock.io/max-iops |
qos_rw_mbytes |
max_mbytes_per_sec |
storage.simplyblock.io/max-mbytes-per-sec |
qos_r_mbytes |
max_read_mbytes_per_sec |
storage.simplyblock.io/max-read-mbytes-per-sec |
qos_w_mbytes |
max_write_mbytes_per_sec |
storage.simplyblock.io/max-write-mbytes-per-sec |
A StorageClass is assigned to a storage pool by three labels, storage.simplyblock.io/namespace,
storage.simplyblock.io/cluster, and storage.simplyblock.io/pool, instead of being named after the pool. A pool
may have any number of classes. The parameter dhchap_node_label is renamed to dhchap_node_selector, and the
node label it names changes from simplyblock.io/pool.<namespace>.<cluster>.<pool> to
storage.simplyblock.io/storage-pool.<pool-uuid>.
Other Changes in 26.4.0-beta1
- The CSI driver no longer passes on the
StorageClassparameterscompression,replicate, andpriority_class. - The default filesystem of a volume is
xfs, and the image pull policy of the storage nodes and the CSI driver defaults toAlways. - The
StorageClusterfieldsspec.storageNodes.dataInterfacesandspec.enableNodeAffinitycannot be changed once set.
Changes in 26.4.0-beta2
- The metrics endpoint of the storage node proxy no longer requires authentication.
- The csi-addons sidecar is deployed only when
enableCSIAddonsis set on theSimplyblockDriver. A deployment that uses csi-addons volume replication has to set it. - The bundled
csi-driver-host-pathand itslocal-hostpathStorageClassare removed from the Helm chart, along withcontrolplane.csiHostpathDriver.enabled.
Changes in 26.4.0-beta3
- The maximum number of NVMe-oF subsystems per storage node is raised from 75 to 500
(Limits). Each subsystem reserves huge page memory, so
maxSubsystemCountshould reflect the planned number of volumes. - The TLS settings of the installation are read from
ControlPlane.spec.source.local.tls, which cannot be changed once set. The environment variablesSB_TLS_SERVE,SB_TLS_CONNECT,SB_TLS_CLIENT_AUTH, andFDB_TLS_*are removed from the operator. - The initial discovery run is an
OperatorOpswith the actionDiscover, and the Helm valuesdiscovery.*are the fields of itsspec.discover. Thesimplyblock-bootstrapConfigMap is no longer created. - The control plane workloads and the CSI controller run on the pod network instead of the host network.
enableArbitrationis passed to the control plane. A two-node cluster that set it with an earlier release runs with arbitration after the upgrade.- Container images are accepted from any registry.
Upgrade Considerations
- TODO, to be confirmed once the
simplyblock-upgradetool is released.
Known Issues
- TODO, to be re-checked against the 26.3 known issues.
Features to Expect with Next Major Release
- The ability to use different erasure coding schemas within the same cluster.