Skip to content

Storage Network

A simplyblock storage node serves its volumes over NVMe/TCP from its SPDK pod. Where that traffic flows is decided once per storage cluster: on the host's own interfaces, on the pod network of the cluster's CNI, or on dedicated interfaces that Multus attaches to the SPDK pod.

The storage network is part of the cluster declaration: spec.cluster.network in the ClusterDeploymentConfig, which becomes spec.storageNodes.network in the StorageCluster. It is fixed when the cluster is created. Changing it later means deleting the storage cluster and creating it again, so the choice is made during planning, together with the Network requirements.

Availability

The spdkOnHostNetwork, podNetwork, and data fields need a Simplyblock Operator and control plane release that includes them. A cluster that declares no network keeps the host-network behavior of earlier releases.

The page follows the order of the decision:

The data ENI on Amazon EKS has a page of its own, Amazon EKS.

Overview

Network Topologies

The SPDK pod sits in one of three network topologies. Each of them exists for a different reason, and the reason decides which one fits a deployment.

SPDK on the host network. The SPDK pod runs in the host's network namespace, and SPDK binds the storage interface directly. The pod is placed on the host network for one reason only: to take hold of a specific high-speed interface. This fits storage nodes with several interfaces of different speeds, where the fastest one is reserved for storage traffic. No CNI, Service, or virtual interface sits between SPDK and the NIC, so this path performs best. This topology is option 1.

SPDK on the pod network. The SPDK pod runs on the pod network of the cluster's CNI, like any other pod. A Kubernetes Service is placed in front of every SPDK pod, and the IP address of that Service is the static storage address of the node. Nothing has to be prepared on the hosts, and no second interface is needed. This fits wherever the bandwidth is not tied to a particular interface. On Amazon EC2, for example, the network bandwidth is set by the instance type and not by the ENI, so a dedicated interface adds no bandwidth. This topology is option 2.

SPDK on the pod network with Multus. The SPDK pod keeps its pod-network interface, and Multus attaches a second interface for the storage traffic. This is the topology for allocating a storage interface to the SPDK pod the Kubernetes-native way, instead of placing the pod in the host's namespace. It requires a secondary interface on every storage node, which on Amazon EKS is a secondary ENI. How the interface reaches the pod depends on the CNI plugin of the NetworkAttachmentDefinition, here called a definition:

  • The host-device plugin (option 3): The entire interface is moved into the SPDK pod. It disappears from the host, so ip addr on the host no longer lists it. The result matches the host network topology, a dedicated high-speed interface for SPDK, allocated the Kubernetes-native way.
  • The macvlan plugin (option 4): A virtual interface on a host interface, with its own routable IP address and its own MAC address.
  • The ipvlan plugin (option 4): A virtual interface on a host interface, with its own routable IP address and the MAC address of its parent.
  • SR-IOV (option 5): A virtual function of an SR-IOV NIC, allocated to the SPDK pod by a device plugin and switched in the NIC's hardware.

Workarounds required

The Multus topology does not work out of the box. The host has to be given a path to the storage addresses, and the routing has to keep every packet on the interface that owns its source address. All of these workarounds are collected in Workarounds for Attached Networks.

The Options at a Glance

Option Declared as Multus Storage traffic flows over Typical use
1. Host network spdkOnHostNetwork: true No The host's own NICs, used directly by SPDK Bare metal with dedicated storage NICs, RDMA
2. Pod network podNetwork: true No The CNI's pod network, behind a Service per storage node One NIC, no host preparation, any cloud
3. Dedicated interface podNetwork: true, data Yes A NIC, port, VF, or cloud interface moved into the SPDK pod A spare port per SPDK pod, a second ENI on Amazon EKS
4. macvlan or ipvlan podNetwork: true, data Yes A macvlan or ipvlan interface on a storage NIC, a VLAN, or the CNI's NIC A storage NIC, or one NIC shared with the CNI
5. SR-IOV virtual functions podNetwork: true, data Yes A VF allocated to the SPDK pod by a device plugin SR-IOV NICs, the SR-IOV Network Operator on OpenShift

Choosing an Option

The decision follows from three questions: whether SPDK may share the host's network namespace, how many NICs the storage nodes have, and what the network accepts.

  1. RDMA is required: option 1. Attached networks carry NVMe/TCP only.
  2. The storage nodes have dedicated storage NICs, and SPDK may run in the host's network namespace: option 1. No CNI, Service, or virtual interface sits between SPDK and the NIC, so this path has the lowest overhead.
  3. The storage nodes have SR-IOV NICs, and SPDK has to stay out of the host's network namespace: option 5. The VF gives SPDK a hardware interface, and the NIC's embedded switch carries the traffic to the host.
  4. The storage nodes have a spare storage NIC, and SPDK has to stay out of the host's network namespace: option 4, or option 3 when the network routes between the storage subnet and the hosts. A routed path is acceptable only where Production Recommendations allow it.
  5. The storage nodes have one NIC, and storage has to stay off the CNI's datapath: option 4, on a VLAN of the CNI's NIC when the switches can trunk a VLAN to the nodes, on the CNI's NIC itself otherwise.
  6. Nothing can be prepared on the hosts, or the bandwidth is capped by the infrastructure anyway: option 2.
  7. The cluster runs on Amazon EKS with a second ENI: option 3, as described in Amazon EKS.

Options 3 to 5 need the network to be prepared before the storage cluster is created (see Requirements of Every Attached Network).

None of the options adds bandwidth. On a virtual machine whose network bandwidth is set by its instance type, that limit applies to the instance as a whole, whichever option carries the traffic.

Production Recommendations

The Network page sets the requirements of a production storage network: a redundant network or two VLANs with NVMe multipathing, at least 10 GBit/s per storage node, no software-defined switch, and no gateway, firewall, or proxy above layer 2 on the storage path. Applied to the options, they lead to these recommendations:

  • Option 1 on a bond or on two VLANs: the reference production setup. The storage NICs are named per node, the redundancy is provided by the bond or by multipathing, and no software interface is added on the data path.
  • Option 5 where SPDK has to leave the host's network namespace: a VF is switched in the NIC's hardware, so the separation costs no software datapath.
  • Option 4: macvlan and ipvlan are kernel interfaces on a physical parent, not a bridge or a virtual switch. They need a host-side sibling interface on every node, which is node configuration and has to be managed as such (see The Host-Side Sibling Interface). Their throughput is to be measured against option 1 on the target hardware before production use.
  • Option 3 with a routed path: the path from the hosts to the SPDK pods runs through a gateway, which the network requirements rule out on the storage path. On bare metal, option 3 is suitable where the hosts reach the storage subnet on-link through another interface. In a cloud network that routes every packet anyway, such as an Amazon VPC, the restriction does not apply in the same way.
  • Option 2: all storage traffic passes the CNI's datapath and shares it with the pod traffic of the worker. It is suitable where the infrastructure caps the bandwidth below what the CNI carries, and for test and edge deployments.
  • MTU: jumbo frames (MTU 9000) end to end on the storage network, on the parent interfaces, the siblings, the definitions, and the switches alike. A mismatch anywhere on the path shows up as stalled connections, not as errors.

Components and Their Networks

The storage network decides the network of the SPDK pod only. The other components of a storage node run on the same network in every option, and the reason for each placement is part of the security model of the deployment.

Component Network Reason
Control plane Pod network It talks to the storage nodes through their agents and SPDK RPC, which the pod network reaches in every option.
Storage node agent Pod network, with the host PID namespace It reads the host's interfaces and opens host connections by entering the network namespace of the host's PID 1. That needs the host PID namespace, but no listener on a host port.
CSI node plugin Host network The NVMe/TCP initiator runs in the host's network namespace, and the kernel reconnects from it after every path loss. A pod-network connect would move to the host on the first reset.
Node preparation Jobs Host network They configure the host (huge pages, kernel modules, devices) and call the agent at its pod address.
SPDK pod Depends on the option Described on this page.

The initiator is always the host. Every option therefore has to give every worker that mounts volumes a path to every storage address, including the storage node on the same worker. The operator checks that path before it reports a node online.

Configuration in Simplyblock

Exactly one of podNetwork and spdkOnHostNetwork is set. data adds dedicated networks to the pod network, so it is only allowed together with podNetwork: true.

Example of the host network (option 1)
spec:
  cluster:
    network:
      spdkOnHostNetwork: true
Example of the pod network (option 2)
spec:
  cluster:
    network:
      podNetwork: true
Example of the pod network with a data network (options 3 to 5)
spec:
  cluster:
    network:
      podNetwork: true
      data:
        - networkAttachmentRef:
            name: storage-data
            addressPool:
              subnet: 10.10.10.0/24
              first: 10.10.10.151
              last: 10.10.10.160
Field Meaning
spdkOnHostNetwork The SPDK pods run on the host network and serve on the NICs each node names in mgmtInterface and dataInterfaces.
podNetwork The SPDK pods run on the pod network. Alone, the pod's eth0 carries all traffic, behind a Service. With data, eth0 carries the RPC and data0 onward carry the volumes.
data[].networkAttachmentRef.name A definition in the namespace of the storage cluster. It is also the network's name in the status of the storage nodes.
data[].networkAttachmentRef.addressPool The range each node's address on the network is taken from. Each node takes the first free address and keeps it for its lifetime, across restarts.
addressPool unset The node takes the address that the interface named in the definition holds on the worker. This suits host-device only.
addressPool.first, addressPool.last Optional, and set together. Both lie inside subnet. Unset, the range is the whole subnet.

The schema enforces these rules, so a declaration that breaks one of them is refused when it is applied:

  • One of podNetwork and spdkOnHostNetwork is true, never both.
  • data needs podNetwork: true, holds up to eight networks, and names each definition once.
  • A cluster with data networks uses the tcp fabric. RDMA is served on the host network only.
  • dataInterfaces are allowed only with spdkOnHostNetwork: true, on the cluster, on the groups of the ClusterDeploymentConfig, and on each StorageNode. A cluster that declares no network keeps the earlier host-network behavior and still accepts them.
  • The network cannot be added, changed, or removed after the cluster is created.

Interface names per node. The NIC names belong to each node, because the workers of a fleet do not always agree on them (eth1 on some, ens6f0 on others). Each group of the ClusterDeploymentConfig names its mgmtInterface and dataInterfaces, and each StorageNode carries them in spec.config, where they are immutable. Workers whose NICs are named differently go into separate groups of the same cluster.

What discovery drafts. Discovery drafts spdkOnHostNetwork: true when most workers have a data NIC besides the management NIC, and podNetwork: true when none does. It adds a note to the draft for either choice, because the network is fixed at creation. It never drafts a data network: options 3 to 5 are an edit of the draft before approval. A group's own networkAttachment field, which discovery drafts when it finds definitions in the namespace, is not deployed. The operator holds such a ClusterDeploymentConfig with the event NetworkAttachmentNotDeployable until the network is declared in spec.cluster.network and the field is removed from the groups.

The draft's network is checked before the draft is approved:

Showing the network of a draft before approval
kubectl -n simplyblock get clusterdeploymentconfig <DRAFT> \
    -o jsonpath='{.spec.cluster.network}{"\n"}'

Option 1: Host Network

The SPDK pod runs in the host's network namespace and serves on the host's NICs. No CNI, Service, or virtual interface lies between SPDK and the NIC. This is the only option that supports RDMA, and the behavior of a cluster that declares no network.

Requirement Reason
A management NIC and one or more data NICs on every storage node SPDK binds the data NICs named for the node. The management NIC carries the control traffic.
For RDMA: RDMA-capable NICs and the RDMA kernel modules The NVMe/RDMA transport needs them on the storage nodes and on the clients.
Host ports free for the SPDK listeners and the RPC on every storage node The listeners bind the host's addresses. See Ports.
Example of the host network with interfaces named per group
spec:
  cluster:
    network:
      spdkOnHostNetwork: true
  nodeSets:
    - name: rack-a
      groups:
        - name: nic-eth
          workers: [worker-1, worker-2]
          mgmtInterface: eth0
          dataInterfaces: [eth1]
        - name: nic-ens
          workers: [worker-3]
          mgmtInterface: ens5f0
          dataInterfaces: [ens6f0]

Each StorageNode takes the names of its group into spec.config.mgmtInterface and spec.config.dataInterfaces.

Option 2: Pod Network

The SPDK pod runs on the CNI's pod network with one interface, eth0, which carries both the RPC and the volume traffic. The pod's own address changes on every restart. The storage node agent therefore keeps a ClusterIP Service in front of each SPDK pod, and the Service address is the address of the node for clients and for the other storage nodes.

Requirement Reason
A CNI that routes pod addresses between nodes Storage nodes and clients reach each other's SPDK pods.
ClusterIP Services reachable from the host's network namespace The initiator connects to the Service address from the host. kube-proxy and the kube-proxy replacement of Cilium both provide this.
No network policy that blocks the storage ports between hosts, SPDK pods, and Services The listeners use the ports the control plane allocates from 4420 upward.
A pod MTU as large as the network allows The pod MTU bounds the NVMe/TCP payload per packet. With an overlay, the pod MTU is the underlay MTU minus the encapsulation, for example, 50 bytes for VXLAN.

Multus is not needed, and nothing is prepared on the hosts: no NIC names, no extra plugin, and no host port. Several storage clusters can therefore share a worker. On Cilium, native routing performs better than a tunnel, because it avoids the encapsulation and the smaller MTU.

The status of each storage node reports the network pod, the interface eth0, and the Service address, with the phase Attached once the pod runs behind its Service.

Attached Networks (Options 3 to 5)

Options 3 to 5 attach the storage interface to the SPDK pod through a definition. They share the requirements below, and each of them needs one or more of the Workarounds for Attached Networks.

Requirements of Every Attached Network

Preparing the network

Simplyblock does not set up the storage network of options 3 to 5. Installing Multus and the CNI plugins, creating the secondary interfaces, VLANs, VFs, and host-side siblings on every node, creating the NetworkAttachmentDefinition, and configuring the switches are part of preparing the infrastructure. The same applies to making every storage address reachable from every worker that mounts volumes. The operator only uses the definition and checks reachability. A node whose storage address is not reachable stays in Provisioning.

Requirement Reason
Multus, in thick-plugin mode Multus attaches the definition's interface to the SPDK pod.
The CNI plugins the definition uses, in the CNI binary directory of every storage node Multus runs these binaries. host-device, macvlan, ipvlan, static, and sbr are part of the reference CNI plugins. sriov is a separate project.
The definition in the namespace of the storage cluster Multus refuses a reference to another namespace.
static IPAM, and the ips capability declared The operator chooses each node's address and passes it to the pod. An IPAM plugin that allocates at attach time (DHCP, host-local, Whereabouts) returns an address nothing recorded, and it can change on a restart while every client still holds the old one.
No default route in the definition The pod's default route stays on eth0, which carries the RPC and the reach to the cluster.
Every worker that mounts volumes reaches every storage address The initiator runs in the host's network namespace. The host needs an address in the storage subnet, or a route to it. The operator checks the path from every worker and holds a node that a worker cannot reach (see Reaching the Storage Addresses).
The CSI node plugin on every worker, linked to the operator The operator reads interface addresses and checks the reachability through the CSI node plugins.
No two interfaces of a pod in one subnet Two routes for one subnet send replies out of the wrong interface. A definition that has to share a subnet chains the sbr plugin (see Source-Based Routing).

The CNI binary directory is /opt/cni/bin on most distributions. K3s and RKE2 use their own directory below /var/lib/rancher, and Multus has to be installed with that directory and the directory its entries point to mounted.

The reachability check. Once the control plane reports a node online, the operator asks the CSI node plugin of every worker to open a TCP connection to the node's SPDK listener on each data network. The network is recorded as Attached, with hostReachable: true, and the node as Online, only when every worker reaches it. A node that fails the check stays in Provisioning, its network is Unavailable, and the event DataNetworkHostUnreachable names the address and the worker that cannot reach it.

Option 3: Dedicated Interface (host-device)

The host-device plugin moves a whole interface into the SPDK pod, and returns it to the host when the pod ends. The interface can be a physical port, a NIC, a VF named in the definition, or a second cloud interface. SPDK then has an interface whose queues and interrupts no other pod shares.

Requirement Reason
One spare interface per SPDK pod on every storage node, with one name or PCI address on all nodes The plugin moves the whole interface, and the definition names it once. Two SPDK pods on a worker need two interfaces and two definitions.
A path from every host to the storage subnet over another interface While the pod runs, the host has no interface on the storage network. The initiator on the same worker reaches its own SPDK pod only through another port, a VF, or a router (see Reaching the Storage Addresses).
Without an addressPool: an address on the interface that does not change The node takes the address the interface holds on the host. A DHCP lease needs a reservation, because the host no longer renews it.
Example of a host-device definition (storage-data)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: storage-data
  namespace: simplyblock
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "storage-data",
      "type": "host-device",
      "device": "ens4f1",
      "capabilities": { "ips": true },
      "ipam": { "type": "static" }
    }

device names the interface. When the interface names differ between nodes but the PCI slots agree, "pciBusID": "0000:3b:00.1" names it instead. The network is declared with or without an addressPool.

Caveats. An interface that returns to the host without its address, for example, after its storage node was removed, holds the next node on that worker until the address is restored. With an addressPool, the interface needs no address on the host.

Option 4: macvlan or ipvlan on a Host Interface

The macvlan and ipvlan plugins create a virtual interface for the SPDK pod on a parent interface that stays on the host. The host and several SPDK pods share the parent. macvlan gives each interface a MAC address of its own, and bridge mode lets the interfaces of one parent reach each other. ipvlan shares the parent's MAC address, and l2 mode places the interfaces in the parent's subnet.

The parent, from the strongest to the weakest isolation of the storage traffic, is one of these:

Parent (master) Isolation Typical use
A dedicated storage NIC, for example, eth1 A NIC and a link of its own A spare storage NIC
A VLAN interface on the CNI's NIC, ens18.100 A tagged VLAN, with a subnet of its own One NIC, a VLAN available
The CNI's NIC itself, ens18 A subnet of its own on the node's segment One NIC, no VLAN available
Requirement Reason
A parent interface on every storage node, with one name on all nodes The definition names it as master.
For macvlan: a network that accepts additional MAC addresses Switches with port security, and virtual switches that refuse unknown MAC addresses, drop the traffic. Cloud networks drop it always. ipvlan avoids this, because it shares the NIC's MAC address.
A host-side sibling interface on every node, with the host's storage address A host cannot reach the macvlan or ipvlan interfaces of its own NIC (see The Host-Side Sibling Interface).
An addressPool The parent stays on the host. Without a pool, the SPDK pod would take the parent's address, and the pod and the host would hold the same address.
Example of a macvlan definition (storage-data)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: storage-data
  namespace: simplyblock
spec:
  config: '{"cniVersion":"0.3.1","name":"storage-data","type":"macvlan","master":"eth1","mode":"bridge","mtu":9000,"capabilities":{"ips":true},"ipam":{"type":"static"}}'

For ipvlan, the type is ipvlan and the mode l2. The network is declared with an addressPool in the storage subnet, outside the range of the hosts' own storage addresses, as in Configuration in Simplyblock.

With the host's address moved from the parent to the sibling, the parent carries no address. Discovery then finds no data NIC on the workers and drafts podNetwork: true without a data network. The draft's network is therefore set as above before the draft is approved.

Caveats. All ipvlan interfaces share the MAC address of the parent. A duplicate-address check that relies on one reply per MAC address cannot tell them apart.

Without a Second NIC

On a VLAN of the CNI's NIC. The switches trunk the VLAN to every node, the clients' nodes included. The VLAN interface is created on every node, without an address, by the network configuration of the distribution: netplan, NetworkManager, or systemd-networkd. On Talos Linux, a machine configuration patch creates it without a reboot:

Example of a Talos machine configuration patch for VLAN 100
machine:
  network:
    interfaces:
      - interface: ens18
        vlans:
          - vlanId: 100

With Cilium, the VLAN is added to bpf.vlanBypass. Cilium's programs on the NIC drop tagged frames of VLANs they do not know, and an agent that started before the VLAN interface existed drops its frames until it restarts, even with the bypass set:

Allowing VLAN 100 through Cilium
helm upgrade cilium cilium/cilium \
    --namespace kube-system \
    --reuse-values \
    --set "bpf.vlanBypass={100}"
kubectl -n kube-system rollout restart ds/cilium

The MTU of the VLAN is bounded by the NIC's MTU. Cloud networks do not carry customer VLAN tags, so a VLAN is limited to networks under the operator's control.

On the CNI's NIC, untagged. No switch configuration and no Cilium setting are needed. The storage subnet is one that no other system on the segment uses, or a range of the node subnet reserved from DHCP. Storage shares the broadcast domain of the node network and is separated from it only by its subnet.

Option 5: SR-IOV Virtual Functions

The SR-IOV Network Device Plugin advertises the VFs of the nodes as an extended resource. A definition annotated with that resource takes the VF the plugin allocates to the SPDK pod, and the SPDK pod requests one of it. The host keeps the physical function or another VF for its own storage address. The NIC's embedded switch carries the traffic between the host and the SPDK pod, on the same node as well, so no host-side sibling is needed.

Requirement Reason
VFs created on the physical function of every storage node (sriov_numvfs) The device plugin advertises existing VFs. It creates none.
The SR-IOV Network Device Plugin, with a pool that selects the storage VFs The plugin allocates a VF to each SPDK pod and tells Multus which one.
For the sriov type: the sriov CNI plugin in the CNI binary directory Neither Multus nor the reference CNI plugins include it.
An address of the host in the storage subnet, on the physical function or another VF The initiator on every node reaches the storage addresses.
VFs in kernel mode, not bound to vfio-pci The SPDK pod uses the VF as a kernel interface for NVMe/TCP.

On OpenShift, the SR-IOV Network Operator provides the VFs, the device plugin, and the sriov plugin, and generates the definition from a SriovNetwork resource. The SriovNetwork sets ipam to '{"type": "static"}' and capabilities to '{"ips": true}', so that the generated definition meets the requirements above.

The definition carries the resource in the k8s.v1.cni.cncf.io/resourceName annotation and names no device:

Example of an sriov definition (storage-sriov)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: storage-sriov
  namespace: simplyblock
  annotations:
    k8s.v1.cni.cncf.io/resourceName: simplyblock.io/storage_vf
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "storage-sriov",
      "type": "sriov",
      "vlan": 100,
      "spoofchk": "on",
      "trust": "off",
      "capabilities": { "ips": true },
      "ipam": { "type": "static" }
    }

vlan, spoofchk, and trust are optional. The sriov plugin sets them on the VF through the physical function.

Where the sriov plugin is not available, or the nodes see only VFs, a host-device definition with "capabilities":{"deviceID":true,"ips":true} and the same annotation attaches the allocated VF.

The network is declared with an addressPool. The VF is unknown until the device plugin allocates it, so no address can be read from it beforehand, and the operator holds a node with an event that says so when the pool is missing. The operator reads the annotation when it checks the definition, records the resource on the node as status.network[].resource beside the assigned address, and passes it to the control plane with the network, which requests one of it for the SPDK container. Nothing else is configured for the resource.

Workarounds for Attached Networks

A storage interface inside the SPDK pod is not visible to the host the way the host network topology is, and the Linux kernel and the cloud networks add constraints of their own. Options 3 to 5 therefore need one or more of the workarounds below. All of them are part of preparing the infrastructure and are not applied by simplyblock.

Workaround Problem it solves Needed with
On-link host address The host has no path to the storage subnet once the interface is in the SPDK pod Options 3 and 5
Host-side sibling interface A host cannot reach the macvlan or ipvlan interface of its own parent Option 4
Routed path The hosts reach the storage subnet only through a router, and the SPDK pod's default route is eth0 Option 3, where nothing on-link is left
Source-based routing Two interfaces of the SPDK pod in one subnet send replies out of the wrong interface Any option whose data interface shares a pod subnet

On Amazon EKS, the routing depends on whether the data ENI sits in the node subnet or in a subnet of its own. Both layouts are described in Amazon EKS.

Reaching the Storage Addresses

The initiator of every worker connects from the host's network namespace, so every host needs a path to every storage address, the SPDK pod on its own worker included. Without a path, the reachability check fails on the workers that have none, and the nodes stay in Provisioning. The path is an on-link host address, a host-side sibling interface, or a routed path.

The host keeps an interface in the storage subnet: a second port, the physical function or another VF, or a VLAN interface. Its address lies outside the addressPool. No route is needed, and the traffic stays on layer 2, as the Production Recommendations require.

Routed Path

Where the hosts reach the storage subnet only through a router, the SPDK pod needs a route back to the hosts' subnet, because its default route is eth0. The route goes into the static IPAM block of the definition, and every worker gets a route to the storage subnet:

Example of a route back to the hosts' subnet in the definition's IPAM block
"ipam": {
  "type": "static",
  "routes": [ { "dst": "192.168.10.0/24", "gw": "10.10.10.1" } ]
}
Adding a route to the storage subnet on each worker
ip route add 10.10.10.0/24 via 192.168.10.1 dev eth0

A router on the storage path is acceptable only where the Production Recommendations allow it.

The Host-Side Sibling Interface

Option 4 gives the SPDK pod a macvlan or ipvlan interface on a parent interface of the host. The Linux kernel does not deliver traffic between a parent interface and its own macvlan or ipvlan interfaces. A host whose storage address sits on the parent therefore reaches the SPDK pods of other nodes, but not the one on its own node. Without a sibling, the reachability check fails from each node to its own SPDK pod, and the node stays in Provisioning with the event DataNetworkHostUnreachable. A client on the storage node would otherwise lose the local path of its volumes and, with it, a failover path.

The solution is a second interface of the same type on the same parent, the sibling, which carries the host's storage address. Two interfaces of one parent reach each other. The parent itself keeps no address in the storage subnet. The sibling is node configuration: it is created on every node, the clients' nodes included, and it has to survive a reboot.

In the examples, the parent is eth1, the sibling is shim0, and IP is the node's own storage address, which differs per node.

NetworkManager keeps the sibling across reboots.

Creating a persistent macvlan sibling with NetworkManager
IP=10.10.10.62
nmcli con add type macvlan con-name shim0 ifname shim0 \
    macvlan.parent eth1 \
    macvlan.mode bridge \
    macvlan.tap no \
    802-3-ethernet.mtu 9000 \
    ipv4.method manual \
    ipv4.addresses $IP/24 \
    ipv6.method disabled \
    connection.autoconnect yes
nmcli con mod eth1 \
    ipv4.method disabled \
    ipv4.addresses "" \
    802-3-ethernet.mtu 9000
nmcli con up eth1
nmcli con up shim0
Removing the macvlan sibling and restoring the address on eth1
nmcli con delete shim0
nmcli con mod eth1 ipv4.method manual ipv4.addresses $IP/24
nmcli con up eth1

The ip commands take effect at once but do not survive a reboot. A reboot, or nmcli con up eth1, puts the address back on eth1, where it clashes with the sibling. A persistent setup runs them from a systemd unit or a network script at boot.

Creating a sibling with iproute2
IP=10.10.10.62
# macvlan: type macvlan mode bridge; ipvlan: type ipvlan mode l2
ip link add shim0 link eth1 type ipvlan mode l2
ip link set shim0 mtu 9000 up
ip addr flush dev eth1
ip addr add $IP/24 dev shim0
Removing the sibling and restoring the address on eth1
ip link del shim0
ip addr flush dev eth1
nmcli con up eth1
Checking the parent and the sibling on a node
ip -br addr show eth1
ip -br addr show shim0

The parent shows no IPv4 address, and the sibling shows the node's storage address.

Source-Based Routing

A pod with two interfaces in one subnet has two routes for that subnet. The kernel picks one route per destination and sends the packet out of that interface, whatever interface the request arrived on. A switch usually passes such a packet on anyway. A cloud network drops it, because an interface may only send packets whose source is one of its own addresses.

The sbr plugin, chained behind the plugin that creates the data interface, moves the data interface's routes into a table of their own. That table is selected only for packets whose source is the data address. Every other packet uses the main table and leaves through eth0. The definition and the failure it prevents on Amazon EKS are shown in Fix in the SPDK Pod: Source-Based Routing.

Verification

The operator reports the network of each storage node:

Showing the network status of the storage nodes
kubectl -n simplyblock get storagenode \
    -o custom-columns=NODE:.spec.workerNode,PHASE:.status.phase,NET:.status.network[*].network,STATE:.status.network[*].phase,ADDR:.status.network[*].address,REACH:.status.network[*].hostReachable

Every node reaches Online. Each data network is Attached, with hostReachable set to true and the address taken from the pool, or adopted from the interface. On the pod network, the network is pod and the address is the Service address. On option 5, the network also carries resource, the device plugin resource the SPDK pod requests.

The paths of a volume on a client node show whether the local storage node is reachable. On the host, or in a privileged pod on the client's node:

Listing the paths of the volumes on a node
nvme list-subsys

Each volume has one live path per storage node that serves it, and the path to the storage node on the same worker is present and live.

Troubleshooting

Symptom Cause Solution
DataNetworkHostUnreachable from each worker to the storage node on the same worker macvlan or ipvlan with the host's storage address on the parent interface Add the host-side sibling and remove the address from the parent.
DataNetworkHostUnreachable from every worker, with host-device The hosts have no interface on the storage network and no route to it Give the hosts an on-link path through another interface, or a route and a return route.
DataNetworkHostUnreachable naming one worker That worker has no address in the storage subnet or no route to it, or a firewall blocks the storage ports Give the worker an address or a route, and allow the ports from 4420 upward.
NetworkAttachmentNotDeployable on the ClusterDeploymentConfig A group carries the networkAttachment field that discovery drafted Declare the network in spec.cluster.network, and remove the field from the groups.
The ClusterDeploymentConfig or StorageCluster is refused with dataInterfaces are allowed only with ... A group or the cluster names dataInterfaces with podNetwork Remove dataInterfaces, or use spdkOnHostNetwork: true.
Tagged frames do not arrive on a node, and Cilium's monitor reports VLAN traffic disallowed by VLAN filter The VLAN is not in bpf.vlanBypass, or the Cilium agent started before the VLAN interface existed Add the VLAN to bpf.vlanBypass and restart the Cilium agent on the node.
The SPDK pod stays Pending with Insufficient simplyblock.io/storage_vf The device plugin advertises no free VF on the node Check the node's allocatable resources, the VFs, and the plugin's pool selectors.
NetworkAttachmentFailed, the definition "names no device or PCI address" A device-plugin definition without an addressPool, or a host-device definition without device or pciBusID Declare an addressPool, or name the interface in the definition.
NetworkInterfaceNoAddress No addressPool, and the interface named in the definition has no usable IPv4 address on the host Assign an address to the interface, or declare an addressPool.
The SPDK pod does not start, and Multus reports Link not found The master or device named in the definition does not exist on the node Create the interface, for example, the VLAN interface, on every storage node.
Every pod on a storage worker fails to start after Multus is installed on K3s or RKE2 Multus cannot run the plugins, because the CNI binary directory or the directory its entries point to is not mounted Install Multus with the distribution's CNI directories mounted into its pod.
No traffic reaches a macvlan interface The switch or virtual switch drops unknown MAC addresses Allow additional MAC addresses on the ports, or use ipvlan, option 4 on a VLAN, or option 5.