Five Networks for a Private Cloud, Not One

The cluster is healthy. One disk fails on a storage node, recovery starts on its own, and within a minute every instance in the cloud slows to a crawl. No service is down. No alert fires beyond the expected one. The rebuild traffic and the tenant traffic are sharing a cable.

That is the failure mode a flat network guarantees, and it arrives at the worst possible moment: while the cluster is degraded and the redundancy is already reduced.

The five networks, and what each one carries

A sixth path carries north-south traffic: the provider network on which floating addresses live. It is usually a set of tagged VLANs rather than a single network, and it is the only one exposed outside the platform.

Four of these are easy to justify. The fifth is the one that gets merged, and it is the one that matters most.

The separation nobody expects: storage public and storage cluster

Both networks carry Ceph traffic, so treating them as one looks reasonable. The arithmetic says otherwise.

What replication actually costs

When a client writes an object, it sends it once, over the public network, to the primary OSD. That OSD then replicates it to the other copies, over the cluster network. With a replication factor of three, one client write becomes two internal writes.

client write of X bytes, replication factor N:

  public network  :  X
  cluster network :  X * (N - 1)

at N = 3, the cluster network carries twice the write
volume of the public network, in normal operation

Reads behave differently: they are served by the primary OSD and never touch the cluster network. So the ratio between the two networks depends entirely on the read-write mix of the workload, and a write-heavy platform loads the cluster network far harder than the public one.

What recovery does to that ratio

Lose an OSD and the cluster re-replicates every placement group it held. That traffic is generated by the cluster itself, runs entirely on the cluster network, and is completely independent of what clients are asking for. It does not wait for a quiet period.

On a merged network, this creates a choice with no good answer. Let recovery run at full speed and client latency collapses. Throttle it, and the cluster stays degraded for longer, which means a longer window during which a second failure causes data loss. Ceph provides the throttles precisely because operators need them on undersized networks.

Separating the two networks removes the choice rather than arbitrating it. That is the entire argument, and it is why network and physical infrastructure design belongs at the start of a platform project rather than at the end.

The fifty bytes that decide overlay throughput

Instance traffic between hypervisors is encapsulated, and encapsulation costs header space. The exact cost depends on the protocol.

physical MTU 1500  :  VXLAN 1450  |  Geneve 1442
physical MTU 9000  :  VXLAN 8950  |  Geneve 8942

VXLAN adds 50 bytes over IPv4; Geneve, which OVN uses by default, adds 58. If an instance believes it has 1500 bytes available while the path carries 1450, large packets are fragmented or dropped depending on the flags. Small requests succeed, bulk transfers stall, and the symptom looks like an application defect.

Encapsulation also costs processor time. Without hardware offload on the network adapters, the hypervisor encapsulates and decapsulates every frame in software, and measured throughput on an overlay network commonly falls well short of the physical link speed even when the cabling is correct. Traffic that leaves through the provider network, being tagged rather than encapsulated, does not pay that cost. This is worth knowing before attributing a disappointing benchmark to the switches.

The storage networks want a large MTU for a different reason: Ceph moves objects, not packets, and a 9000-byte frame carries six times the payload of a 1500-byte one for the same per-packet processing cost.

The trap is that MTU has to be raised on every hop. Server interface, bond, VLAN sub-interface, and every physical switch port in the path. One port left at the default produces exactly the intermittent behaviour described above, and it is invisible in any configuration review that does not include the switches.

ip link show

ping -M do -s 8972 STORAGE_PEER_ADDRESS

The second command is the one that settles it. With the do-not-fragment flag set and a payload of 8972 bytes, a 9000-byte path succeeds and anything smaller fails outright. The payload figure accounts for the 20-byte IP header and the 8-byte ICMP header.

Management has to survive everything else

The management network is how the platform gets repaired. If it shares a physical path with storage recovery, it saturates at the exact moment an operator needs to log in, and a routine incident becomes an unreachable cluster.

This is the one separation that is not about throughput. It is about keeping a working door into the building while the building is on fire. It deserves its own physical path, or at minimum strict traffic priority on a shared one.

What merging two networks actually costs

Three of the five reveal themselves only when something else has already gone wrong. That is what makes them expensive: the design flaw and the incident arrive together, and the incident gets the blame.

Sizing the links

The starting rule is that the cluster network should be at least as fast as the public one, and preferably faster, because write traffic is multiplied on it while read traffic is not.

Link aggregation deserves one caution. Bonding increases aggregate capacity, not the speed of any single connection: a given TCP flow is hashed onto one member link and cannot exceed it. Ceph benefits substantially, because it opens many connections between many OSDs. A single live migration does not, because it is one flow. Sizing a migration window on the aggregate figure produces a schedule that will not be met.

A verification sequence

1. List every interface, bond and VLAN sub-interface with its MTU, on every node, and confirm they agree.
2. Run the do-not-fragment test between two storage nodes, in both directions.
3. Confirm that Ceph is actually configured with a separate cluster network, rather than assuming the cabling implies it.
4. Confirm the switch port MTU on every port in the storage path, including uplinks between switches.
5. The decisive one. Take one OSD out on purpose, during a maintenance window, and measure client latency while recovery runs. If it moves, the separation is not effective, whatever the configuration says.

That last check is the only one that tests behaviour rather than configuration, and it is the same principle that applies to unified Ceph storage generally: the cluster tells you the truth when something is removed, not when everything is running.

What this article does not cover

Bonding modes, switch stacking and spine-leaf fabric design are decisions of their own, and they depend on hardware that varies from one site to the next. Encryption in transit is deliberately left out because it changes the arithmetic above: it adds processing cost per byte on both storage networks, and that cost has to be measured on the actual hardware rather than estimated.

Hardware offload paths and kernel bypass techniques are out of scope. They matter at high throughput and they change how the overlay behaves, but they are irrelevant until the segmentation itself is correct.

Sources

• Ceph documentation, Network Configuration Reference — public and cluster network roles, replication and recovery traffic.
• OpenStack Neutron documentation, MTU considerations — encapsulation overhead and per-network MTU configuration.
• OpenStack Neutron documentation, OVN migration — Geneve overhead compared with VXLAN and GRE.
• RFC 8926, Geneve: Generic Network Virtualization Encapsulation — header structure.