On a 2.27 GB disk, converting the image to qcow2 took about ten seconds. Adapting the guest so it would boot on the new platform took about five minutes, and applying its network configuration took another minute or two. The data copy was three percent of the work.
That ratio is the single most useful thing to know when planning a cold migration window, and it is the opposite of how most people estimate one. Migration schedules are usually built on disk size and network throughput, which is the part that finishes first. This article walks through what actually happens between shutting down a source VM and powering on its replacement, where the time goes, and the three things that break migrations in practice.
A disclosure before anything else: the figures and the failure cases here come from Yonder, our own cold-migration platform, and from the estates we run it on. We say so plainly rather than presenting them as neutral measurements. The mechanics described are not specific to it — anyone driving virt-v2v by hand meets the same phases and the same constraints.
Scope
- Cold migration only: the source VM is shut down for the duration. Live migration between different hypervisor platforms is a different problem with different trade-offs.
- Sources considered: VMware vSphere and ESXi, OpenStack. Targets: OpenStack, Proxmox VE, KubeVirt and OpenShift Virtualization, Nutanix AHV.
- Linux guests unless stated. Windows guests differ mainly in driver handling, noted where relevant.
- Figures are measurements from specific hosts, not benchmarks. Yours will differ; the ratio between phases is what transfers.
The five phases
Every cold migration, whatever tooling drives it, is the same sequence.
The source VM is shut down. Its disks are acquired, in a way that depends entirely on the source platform. The disks are converted to a format the target accepts, normally qcow2. The guest is adapted so it can boot on hardware it has never seen. The result is uploaded or imported on the target, and the new VM is powered on.
Only the third phase is a data operation. The others are platform mechanics, and phase four is where the window is actually spent.
Acquiring the disks is platform-specific, and it matters
On VMware, the safe path is a consistent snapshot followed by reading the disks over NFC on HTTPS. The snapshot is what makes the read coherent; without it you are copying a filesystem that may still have been written to.
On OpenStack the answer depends on how the instance stores its root disk, and getting this wrong is how you damage a source you meant to leave alone. A volume-backed instance should be exported through a temporary snapshot of the volume, so the original Cinder volume is never touched. An instance with an ephemeral root goes through an instance snapshot instead. A mixed layout needs both, applied per disk.
The rule underneath: a cold migration must be non-destructive on the source. If the rollback plan is "power the original back on", then nothing in the acquisition phase may modify it. That constraint eliminates several shortcuts that look faster.
Conversion is the fast part
Converting a raw or VMDK disk into qcow2 is an I/O-bound operation that modern hardware handles quickly. In the measurement above, ten seconds for 2.27 GB. It scales roughly linearly with size and with the speed of your working storage, and it is the phase people optimise because it is the one they expect to dominate.
qemu-img convert -p -O qcow2 disk.vmdk disk.qcow2
It is worth optimising the storage it runs on, but for a different reason: the working directory is a common cause of outright failure. See below.
The guest fixup is the slow part, and here is why
A VM that booted on VMware will not necessarily boot on KVM. Its initramfs contains the storage drivers it was built with, its network interfaces are named after hardware that no longer exists, and its bootloader may reference a controller that is not there. Adapting all of this is what virt-v2v-in-place and virt-customize do.
virt-v2v-in-place -i disk disk.qcow2
The way they do it explains the cost. These tools boot a small purpose-built virtual machine — the libguestfs appliance — attach the disk image to it, and work on the filesystem from inside that appliance. It is a full guest boot for every disk you adapt.
If the host running the conversion has no access to /dev/kvm, that appliance runs under software emulation. Emulating a guest boot is roughly an order of magnitude slower than running it with hardware assistance, which is where the 300 seconds against 10 seconds comes from. On a migration of two hundred VMs, that difference is measured in days.
Hardware acceleration, and why any doubt should resolve to emulation
The obvious fix is to give the conversion process access to /dev/kvm. It is the right fix, and it has two failure modes that are worth knowing before you rely on it.
Permitted is not the same as working. We measured a host where the device was mapped and readable, where libguestfs reported qemu KVM: enabled, and where QEMU then aborted while starting the appliance because the kernel refused a write to model-specific register 0x345. The same configuration worked on a host with a newer kernel. On the first host, forcing acceleration did not make the fixup slower; it made it fail.
That is why a device check is not sufficient and a functional probe is. Boot an appliance and see whether it comes back:
docker run --rm --user 1001 --group-add <GID> --device /dev/kvm:/dev/kvm \
--entrypoint "" <conversion-image> \
sh -c 'LIBGUESTFS_BACKEND_SETTINGS=force_kvm libguestfs-test-tool >/tmp/t.log 2>&1; \
echo "rc=$?"; grep -iE "qemu KVM:|failed to set MSR|Assertion" /tmp/t.log'
A return code of zero and a line reporting KVM enabled means the appliance boots. Anything else means this host cannot accelerate the fixup, and the correct answer is to leave it in emulation rather than to have migrations fail unpredictably.
Group membership is not optional. On a typical host /dev/kvm is owned root:kvm with mode 0660. A process that is neither root nor in the kvm group gets EACCES on it. Mapping the device into a container that runs as an unprivileged user, without adding the group, produces the worst possible outcome: no error you will notice, a silent fall back to emulation, and the belief that acceleration is in place. Check the numeric group id on the host and pass it explicitly:
stat -c 'owner=%U group=%G gid=%g mode=%a' /dev/kvm && getent group kvm
A related packaging point: a hard device mapping in your base configuration makes the container refuse to start on any host that lacks /dev/kvm. Keep it in an optional overlay so the same configuration runs on hosts that cannot accelerate.
Import: every target pulls, and that is a network constraint
The last phase is where a design assumption catches people out. On several targets, the disk is not pushed by the migration host — the target pulls it.
On Proxmox VE, the modern path is entirely through the REST API: the image is staged on an import-capable storage with download-url, then materialised with import-from. Both are asynchronous node tasks. The consequence is that the Proxmox node must be able to reach the migration host over HTTP. An intercepting proxy anywhere on that path breaks the import, usually with an error that points at the storage rather than at the proxy.
On Nutanix AHV, the image is registered in Prism Central from a URL source, which means Prism Central pulls it. Same requirement, different component. Note also that the v4 vmm API is Prism-Central-only, so a standalone Prism Element is not a valid endpoint, and it needs pc.2024.3 with AOS 7.0 or later.
On OpenStack, the image goes through Glance and, for volume-backed instances, becomes a Cinder volume. The intermediate Glance images must be cleaned up after the server reaches ACTIVE, not before — cleaning early can race the boot of a volume-backed instance.
Plan the firewall and proxy rules for this before the migration window, not during it. "The target pulls from the staging host over HTTP" is a sentence that belongs in your network change request.
Three things that break migrations
The SCSI controller has to match the guest, not the target. A Linux guest whose initramfs contains only virtio drivers will not boot on a pvscsi controller — it comes up with no root device. Use virtio-scsi-single for Linux guests. Windows guests are the opposite case and work with pvscsi. This is a per-VM decision, not a per-target one, and picking one setting for a mixed estate guarantees that half of it fails.
Outside OpenStack, there is no port-level IPAM. On OpenStack you can request a fixed IP on the Neutron port at creation and it is simply true. On Proxmox VE, KubeVirt and Nutanix AHV there is no equivalent: a static address has to be written inside the guest, which means the fixup phase has to identify the right interface. The reliable key is the MAC address, propagated from the source. Any migration plan that assumes addresses will follow automatically on these platforms is wrong, and the symptom appears after the VM boots, unreachable.
The working directory is on the root filesystem by default. Conversion writes intermediate qcow2 files somewhere, and containerised tooling defaults to a volume under /var/lib/docker — that is, on /, not on whatever large storage you provisioned for the project. The failure is [Errno 28] No space left on device, several minutes into a migration, after the source has already been shut down. Relocate it onto sized storage, check that the process user can write there, and if it is NFS, check the root squash mapping. A preflight that refuses the migration up front when the working directory is too small is worth more than a retry.
df -h /mnt/migration_temp; stat -c '%U %G %a' /mnt/migration_temp
What this means for a migration window
Estimate per VM, not per terabyte. For each VM, the window is roughly the acquisition time, plus the conversion time which scales with disk size, plus the fixup time which scales with the number of disks and barely at all with their size, plus the import time.
Because the fixup dominates and is per-disk, a VM with four small disks can take longer than a VM with one large one. That is counter-intuitive enough to be worth checking against your actual inventory before committing to a schedule.
Two levers change the arithmetic. Hardware acceleration of the appliance, where the host supports it, cuts the dominant phase by roughly an order of magnitude. Parallelism helps if your working storage and your source platform can sustain the concurrent reads; conversion is I/O-intensive, and running four at once on storage sized for one converts a compute problem into a storage problem.
Measure one representative VM end to end, on your storage and your network, before you extrapolate. A single real migration tells you more than any estimate, and it costs one maintenance window.
What this article does not cover
Live migration between platforms, which trades a shutdown window for a much more constrained set of requirements. Windows guest specifics beyond driver selection, including activation and domain membership. Application-level cutover — DNS, load balancer members, licence servers keyed to hardware identifiers — which is usually where the real complexity of a migration project sits, well above the VM layer. And the inventory and dependency-mapping work that decides migration order, which no tool does for you.
If Proxmox VE is your target and you are configuring a fresh node, the deb822 repository procedure is the first step.
Migrating a handful of VMs is a technical exercise. Migrating an estate is a sequencing problem, and the tooling is the part that matters least. We plan and run platform exits under VMware exit and migration, and design the target platform under infrastructure architecture and deployment. If you take one number from this article, take the ratio: the copy is not the slow part.
