Ceph as the Storage Backend for OpenStack: Users, Keyrings, and Four Traps

You point Glance, Cinder and Nova at the same Ceph cluster. The deployment completes without an error. Then the first instance takes four minutes to boot from a two-gigabyte image, and the first volume backup fails with a permission denied.

Neither of those failures shows up during deployment. That is the whole problem: the configuration that produces them is syntactically valid, the containers start, the API answers. What is wrong only becomes visible when someone actually uses the cloud.

Why a single backend at all

Three services need persistent storage: Glance for images, Cinder for volumes, Nova for ephemeral disks. Pointing all three at one Ceph cluster is not only about consolidating hardware.

The real gain is that a volume created from an image becomes a copy-on-write clone rather than a copy. No bytes move. A two-hundred-gigabyte image produces a bootable volume in about a second, and that volume occupies almost nothing until it is written to. Live migration stops copying disks, because the disk was never on the hypervisor to begin with.

Every one of those benefits has a precondition. None of the preconditions is checked at deployment time.

The pools, and who is allowed to touch them

Four RBD pools, three Ceph clients. The mapping is not one client per service, and that is the first thing to get right.

Pool	Used by	Ceph client
images	Glance images	client.glance
volumes	Cinder volumes	client.cinder
vms	Nova ephemeral disks	client.cinder
backups	Cinder backups	client.cinder-backup

ceph osd pool create images 128
ceph osd pool create volumes 128
ceph osd pool create vms 128
ceph osd pool create backups 64

ceph osd pool application enable images rbd
ceph osd pool application enable volumes rbd
ceph osd pool application enable vms rbd
ceph osd pool application enable backups rbd

Forgetting the application tag does not break anything immediately. It produces a persistent health warning, and it removes the safety net that stops another service from writing into an RBD pool.

The capabilities are the actual configuration

ceph auth get-or-create client.glance \
    mon 'profile rbd' \
    osd 'profile rbd pool=images'

ceph auth get-or-create client.cinder \
    mon 'profile rbd' \
    osd 'profile rbd pool=volumes, profile rbd pool=vms'

ceph auth get-or-create client.cinder-backup \
    mon 'profile rbd' \
    osd 'profile rbd pool=backups'

Read the second one again. The Cinder client is granted access to two pools, one of which belongs to Nova. That is not sloppiness, and it leads directly to the first trap.

Trap one: Nova authenticates as cinder

The natural assumption is that Nova should have its own Ceph client. It does not. In a standard deployment, Nova uses the cinder identity against the vms pool.

The reason is that a clone spans two pools. When an instance boots from a volume, or when an ephemeral disk is created from an image, one client has to read the source object and write the destination object. A clone is not a copy performed by the cluster on its own behalf: it is an operation performed by a client that must hold rights on both ends.

Create a separate client.nova with rights on vms only, and the deployment still succeeds. Volumes still get created. But the cross-pool clone silently stops being possible, and every instance falls back to a full copy. The symptom is slowness, not an error, which is why this one survives so long.

Trap two: cinder-backup needs two keyrings

Backing up a volume means reading from volumes and writing to backups. Those are two different identities: cinder for the source, cinder-backup for the destination. The backup container therefore needs both keyrings, not just its own.

/etc/kolla/config/cinder/cinder-backup/ceph.conf
/etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
/etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring

Ship only the second keyring and the deployment succeeds, the service starts, the API accepts a backup request. The failure appears at the first backup attempt, which in most organisations means it appears in production, weeks later, on a schedule nobody is watching.

Trap three: the keyring you exported will not parse

This one costs an afternoon the first time. Exporting a keyring produces a file that looks correct and is not.

ceph auth get client.glance -o ceph.client.glance.keyring

The resulting file is indented:

[client.glance]
        key = AQBxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=images"

Ceph clients parse this file strictly. Leading whitespace on the key line makes the key unreadable, authentication is refused, and the error message points at credentials rather than at formatting. Nothing in the file looks wrong to a human reader.

sed -i 's/^[[:space:]]\+//' ceph.client.glance.keyring

You will find this line in most deployment guides with no explanation, which makes it look like superstition. It is not. Run it on every exported keyring, and on ceph.conf as well.

Trap four: using Ceph means not enabling Ceph

The deployment flag named after Ceph does not mean what it appears to mean. In Kolla-Ansible, enable_ceph controlled the deployment of a Ceph cluster by the tool — a capability that has since been removed in favour of external clusters managed by cephadm. It never controlled whether OpenStack uses Ceph.

enable_ceph: "no"

glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"

ceph_glance_user: "glance"
ceph_cinder_user: "cinder"
ceph_nova_user: "cinder"

Setting the first flag to yes because you are using Ceph is the natural mistake, and it sends the deployment down a path that no longer exists.

The performance trap nobody warns you about

Everything above is a permissions problem. This one is not, and it is the one that makes people conclude that Ceph is slow.

RBD cloning only works between images in raw format. The Ceph documentation states it plainly: QCOW2 is not recommended for virtual machine disks, and booting from Ceph requires the raw format in Glance.

Upload a QCOW2 image and nothing fails. Instead, every instance created from it triggers a full sequence: download the image to the compute node, convert it to raw locally, upload it back into the cluster as a flat object. A boot that should take seconds takes minutes, the disk is no longer thin, and the same conversion happens again on the next hypervisor.

What the log actually says

The message is explicit once you know to look for it. Nova reports that an RBD image clone requires the image format to be raw, and names the format it found instead. It appears at debug level, which is why it is usually missed.

Converting before upload

qemu-img convert -f qcow2 -O raw source.qcow2 source.raw

openstack image create \
    --file source.raw \
    --disk-format raw \
    --container-format bare \
    --public "Ubuntu 24.04"

The raw file is larger on disk in the image pool. That cost is paid once, and it is recovered many times over because every volume cloned from it takes no space at all until written.

One more setting is required, and it is easy to miss because its absence produces no error: Glance must expose the direct location of its images, otherwise Cinder cannot tell that the source already lives in Ceph and falls back to copying. The relevant options are show_image_direct_url and, depending on the driver, show_multiple_locations.

What a lab does not show you

Training environments commonly run two OSDs on a single node, with replication reduced accordingly:

ceph config set global osd_pool_default_size 2
ceph config set global osd_pool_default_min_size 1

This is legitimate in a lab. The default CRUSH failure domain is the host, so with a single host the cluster cannot satisfy three replicas and stays in a degraded state until the size is lowered. Reducing it is the only way to reach a healthy cluster on one machine.

It is also unacceptable in production, for two independent reasons.

With min_size at 1, the cluster keeps accepting writes when only one copy is reachable. Lose that OSD before the second replica is written and the data is gone, with no warning that anything was ever at risk. With both OSDs on one host, there is no host-level fault tolerance at all: a single reboot takes the whole cluster offline.

Production values are three replicas, a minimum of two, and at least three hosts. The capacity consequences of that choice — and what happens to a three-node cluster when one node is missing — are worked out in our article on sizing a Ceph cluster for production.

A verification sequence

Five checks, in order. Each one answers a question the deployment did not.

1. Cluster health and OSD count: ceph -s should report HEALTH_OK, with every OSD up and in.
2. Capabilities as actually stored: ceph auth get client.cinder must list both the volumes and the vms pool.
3. Image format as Glance sees it: query the image and confirm the disk format is raw, not qcow2.
4. Image present in the pool: rbd -p images ls should contain the image identifier.
5. The decisive one. Create a volume from that image, then ask Ceph whether the new object is a child of the image snapshot: rbd children images/IMAGE_ID@snap. If the volume appears, cloning works. If the list is empty, a full copy happened and everything above needs revisiting.

That fifth check is the one worth automating. It is the only one that tests the behaviour rather than the configuration, and it is the difference between a cloud that boots instances in seconds and one that boots them in minutes. Designing and operating that verification is part of how we deliver unified Ceph storage infrastructure, and it is the first thing we run on an existing private cloud platform that reports slow instance creation.

What this article does not cover

Manila with native CephFS and the RADOS Gateway for object storage both require their own clients, their own pools and, in the gateway case, an authentication integration with Keystone. They are separate subjects.

Placement group tuning, erasure coding for the backup pool, and cluster capacity planning are also out of scope here. So is the network layout, although it matters more than most of what is above: separating the public and cluster Ceph networks is what keeps recovery traffic from starving client traffic during a rebuild.

Sources

• Ceph documentation, Block Devices and OpenStack — pool layout, client capabilities, copy-on-write cloning and the raw format requirement.
• Ceph documentation, cephadm — bootstrap, OSD deployment and pool configuration.
• Kolla-Ansible documentation, External Ceph — backend flags and configuration file placement.
• OpenStack release notes, 2026.1 Gazpacho, April 2026 — current release at the time of writing.