Ceph sizing fails in a predictable way. The cluster is built to hold today's data, it runs well for a year, and then it crosses 75% utilisation and starts warning. At 85% it stops accepting writes. By that point, adding capacity means rebalancing a full cluster under production load, and the hardware order takes six weeks.
The arithmetic that avoids this is not complicated, but it has three steps that most sizing guides skip: raw capacity is not usable capacity, the fill ceiling is not 100%, and the cluster has to stay under that ceiling with one node down. This guide works through each rule, then applies them to a concrete target.
Scope
- Applies to Ceph as deployed under Proxmox VE 8.x and 9.x, and to standalone Ceph clusters.
- Figures are rules of thumb for replicated pools, which is what virtualisation workloads use. Erasure-coded pools follow different arithmetic.
- You will need a target usable capacity and a rough idea of your workload profile before starting.
- No commands here change anything. The verification commands are read-only.
Rule 1: usable capacity is raw capacity divided by three, then by four thirds
A replicated pool with size 3 stores three copies of every object. Forty terabytes of data therefore occupy 120 TB of raw disk. That part is widely understood.
The part that gets missed is the fill ceiling. Ceph raises a nearfull warning at 85% by default and stops accepting writes at 95%, but the practical ceiling is lower. Performance degrades well before the warning, and more importantly, the free space is what the cluster uses to re-replicate data after a failure. A cluster with no headroom cannot heal. Plan for a maximum of 75% steady-state utilisation.
So the full calculation from usable to raw is:
raw = usable x replication / 0.75
For 40 TB usable at replication 3: 40 x 3 / 0.75 = 160 TB raw. Not 120 TB, and certainly not 40 TB.
You can read your current position at any time:
ceph df
The RAW STORAGE block gives total, available and used. The POOLS block gives MAX AVAIL per pool, which already accounts for replication and for the fullest OSD in the pool. MAX AVAIL is the number to watch, not the raw total.
Rule 2: size for the cluster you have after losing a node
This is the step that turns a correct calculation into a wrong cluster. If your cluster is exactly at 75% with every node up, then losing one node out of four puts the survivors at 100%. Writes stop, and the cluster cannot rebuild the missing replicas because there is nowhere to put them.
The requirement is that the surviving nodes stay under the ceiling. With N nodes of equal size:
raw >= usable x replication / 0.75 x N / (N - 1)
The penalty shrinks as the cluster grows, which is one of the real arguments for more, smaller nodes:
| Nodes | Overhead factor | Raw needed for 40 TB usable |
|---|---|---|
| 3 | 1.50 | 240 TB |
| 4 | 1.33 | 213 TB |
| 5 | 1.25 | 200 TB |
| 6 | 1.20 | 192 TB |
| 10 | 1.11 | 178 TB |
A three-node cluster needs six times its usable capacity in raw disk. That is the true cost of the minimum viable Ceph deployment, and it is worth knowing before the budget is fixed rather than after.
Rule 3: three nodes is a floor, not a target
Three nodes is the minimum because Monitors reach quorum by majority and replication 3 needs three distinct failure domains. It works. But a three-node cluster in maintenance is a two-node cluster, and a two-node cluster has no headroom for a second incident.
Two-node deployments are technically possible using an external quorum device. They are not appropriate for production data, and the reason is not the quorum: it is that replication 2 tolerates exactly one failure, and during the rebuild after that failure some objects exist in a single copy. A second disk error during that window is data loss, not degraded service.
Use replication 3 for anything you would be asked to explain the loss of.
CPU: roughly one core per OSD
Each active OSD consumes on the order of one CPU core under normal load, and more during recovery or backfill. A node with 12 OSDs should have at least 16 physical cores, and that figure assumes the node does nothing else.
In a hyperconverged Proxmox VE cluster it does something else: it runs virtual machines. Budget the OSD cores separately from the VM cores and add them. A node with 12 OSDs and 30 VMs is not a 16-core machine.
Clock speed matters more than core count here. Ceph benefits more from fast cores than from many slow ones, because much of the OSD path is latency-bound rather than throughput-bound.
RAM: 4 GB per OSD, and never swap
Two gigabytes per OSD is the documented minimum. Four to eight is the working figure for anything carrying real load, because that memory holds the BlueStore cache, network buffers and internal OSD structures.
The failure mode of under-provisioned RAM is worth understanding, because it does not look like a memory problem. The node starts swapping, OSD operations exceed their timeouts, Monitors mark the OSDs down, and the cluster begins recovering data that was never actually lost. The recovery adds load, which causes more swapping. A node with 12 OSDs wants 32 GB as a floor and 64 to 96 GB to be comfortable, before any VM memory.
Check what your OSDs are actually using:
ceph osd df
The META column shows Ceph's own metadata per OSD. The VAR column shows how far each OSD deviates from the cluster mean, and STDDEV summarises the spread. A VAR near 1.0 across all OSDs means the data is evenly distributed; a wide spread means some OSDs will hit the fill ceiling long before the cluster average does, and MAX AVAIL will drop accordingly.
Disks: dedicated, raw, and never behind a RAID controller
Ceph manages block devices directly through BlueStore, without an intervening filesystem. Each disk gets its own OSD daemon. This is not a preference, it is how the failure detection and recovery logic is built.
Putting hardware RAID underneath is the single most common sizing mistake, and it is worse than merely redundant. The RAID controller hides individual disk failures from Ceph, which means Ceph cannot start recovery when it should. The controller's own cache and write reordering interfere with BlueStore's assumptions. And you pay for the redundancy twice: once in RAID parity, once in Ceph replication. If your controller supports it, use HBA or IT mode and present the disks raw.
On disk class, the trade-off is conventional: NVMe for latency-sensitive workloads, SATA SSD as the general-purpose choice for virtualisation, HDD where sequential throughput and cost per terabyte dominate. Mixed clusters are supported through device classes, and CRUSH rules can pin a pool to a class:
ceph osd crush rule create-replicated fast-rule default host ssd
What matters for sizing is that disk count, not just disk capacity, sets your parallelism. Six 8 TB disks and twelve 4 TB disks give the same raw capacity and very different IOPS.
Network: two networks, 10 GbE as the floor
Ceph generates substantially more network traffic than the client workload alone, because every write is replicated to two other OSDs across the network. Ten-gigabit interfaces are the practical minimum; 25 GbE or faster is appropriate for larger or performance-sensitive clusters.
The public network carries client traffic and inter-daemon communication. The cluster network carries replication and recovery traffic exclusively. Separating them keeps a recovery event from starving your VMs of storage bandwidth, which is exactly when you least want that to happen. Physically separate interfaces are better than VLANs, but VLANs are much better than nothing.
There is a sizing consequence people miss: recovery time is a function of disk size divided by the recovery bandwidth you allow. A single 16 TB disk failing on a 10 GbE cluster that is also serving production is a long rebuild, and the cluster is degraded for its duration. Larger disks are cheaper per terabyte and slower to heal. That is a real trade-off, not a detail.
In a Proxmox VE cluster, Corosync also wants its own low-latency path. Corosync latency spikes cause false failure detection and unnecessary fencing, so it should not share an interface with replication traffic.
Monitors, Managers and metadata servers
These are cheap compared with OSD nodes, and they are sized differently.
| Component | Count | Resources |
|---|---|---|
| Monitor (MON) | 3, odd numbers only | 2-4 cores, 4-8 GB RAM, small SSD |
| Manager (MGR) | 2 minimum | Similar to a Monitor |
| Metadata server (MDS) | Only if using CephFS | 8 GB RAM minimum, 16-32 GB for large filesystems |
Monitors handle cluster metadata only, never data, so they do not need fast disks or dedicated high-speed interfaces. Five Monitors are worth it on large clusters; more than that mostly adds Paxos coordination cost.
MDS sizing is the exception: metadata servers are RAM-bound, because performance depends on holding the working set of filesystem metadata in cache. A filesystem with millions of small files needs a lot more than the default cache limit allows.
Placement groups: the formula, and why you probably do not need it
The classical formula gives a starting point:
PGs per pool = (OSDs x 100) / replication / number of pools
For 25 OSDs, replication 3 and 5 pools: (25 x 100) / 3 / 5 = 167, rounded up to the next power of two, so 256.
Too few PGs concentrate data on a subset of OSDs and limit parallelism. Too many increase OSD memory and CPU cost and lengthen recovery. Both are real, which is why the number used to require care.
The autoscaler now handles this. It is enabled by default on new pools and adjusts PG counts based on actual pool usage. Leave it on unless you have a specific reason not to, and check what it thinks:
ceph osd pool autoscale-status
Knowing the formula is still worth it, because it tells you whether the autoscaler's answer is in the right range.
Worked example: 40 TB usable
Applying the rules in order.
Raw capacity. 40 TB usable x 3 replicas / 0.75 fill ceiling = 160 TB, then the node-loss factor. At five nodes: 160 x 1.25 = 200 TB raw. That is 40 TB raw per node.
Disks. Five 8 TB SATA SSDs per node gives 40 TB, for 25 OSDs across the cluster. Ten 4 TB disks per node would give the same capacity with double the parallelism and faster individual rebuilds, at higher cost per node and more drive bays.
CPU. Five OSDs per node means roughly 5 cores for Ceph. In a hyperconverged setup, add the VM allocation. Sixteen cores per node is a reasonable starting point; 32 if the nodes are also carrying dense VM workloads.
RAM. Five OSDs at 4 GB each is 20 GB for Ceph, plus the host, plus VM memory. Sixty-four gigabytes is a sensible floor for a hyperconverged node of this shape.
Network. Two 10 GbE interfaces minimum, one public and one cluster, plus a separate path for Corosync. With 8 TB disks, 25 GbE on the cluster network materially shortens rebuild windows.
Monitors and managers. Three Monitors and at least two Managers, co-located on three of the five nodes.
The result is a five-node cluster, 25 OSDs, 200 TB raw, 40 TB usable, which survives a full node loss without crossing the fill ceiling. Sizing this to 120 TB raw, which is what "40 TB times replication 3" suggests on its own, would have produced a cluster that reaches its warning threshold on day one.
Five mistakes that cost the most
Sizing to raw capacity divided by replication. It omits the fill ceiling and the node-loss margin, and it understates the requirement by roughly a factor of two.
Hardware RAID under the OSDs. It hides failures from Ceph, interferes with BlueStore, and pays for redundancy twice.
Replication 2 in production. It tolerates one failure, and during the rebuild some data exists in a single copy.
Mismatched nodes. Nodes with very different capacities create permanent imbalance: CRUSH weights by size, so the largest node receives the most data and becomes the constraint on MAX AVAIL for the whole cluster.
One flat network. Recovery traffic and client traffic compete, so the cluster is slowest exactly when it is degraded.
What this guide does not cover
Erasure coding, which changes the capacity arithmetic substantially and suits archival workloads better than virtual machine disks. Multi-site and stretched clusters, where latency between sites becomes the dominant constraint. Detailed BlueStore tuning, which is workload-specific and should follow measurement rather than precede it. And the separate DB and WAL devices that can front slower OSDs, which are worth their own treatment.
If you are configuring repositories on a fresh Proxmox VE 9 node before any of this, the deb822 repository procedure comes first.
Sizing is where a Ceph cluster is made cheap or expensive, and the decision is effectively permanent once the hardware is bought. We size, deploy and operate distributed storage under unified storage infrastructure, and review existing architectures under IT audit and architecture consulting. If you have a target capacity and a workload profile, the arithmetic above will get you most of the way there on your own.