How to Migrate Virtual Machines Between Hosts: Live VM Migration in KVM

Migrating virtual machines between hosts is a critical operation for ensuring high availability, load balancing, and infrastructure maintenance. In this article, we will take a detailed look at live migration in a KVM (Kernel-based Virtual Machine) environment. We will explore the necessary conditions, tools, and step-by-step instructions so that you can safely and efficiently move virtual machines between your hosts without interrupting their operation. You will learn how to configure the network, check compatibility, and perform the migration using the command line and graphical tools.

Contents:

• Preparing for Migration
• Configuring the Network for Migration
• Checking Host Compatibility
• Performing Migration with virsh
• Troubleshooting

Preparing for Migration

Before starting the virtual machine migration, you need to perform a number of preparatory steps to ensure a successful and seamless move. Key aspects of preparation include checking resources, setting up shared storage (if necessary), and configuring access rights. Proper preparation minimizes the risk of problems during migration and ensures minimal downtime for your virtual machine.

Checking Resource Availability

Make sure the target host has enough resources (CPU, RAM, disk space) to accommodate the virtual machine. Lack of resources can lead to performance issues or even migration failure. Check CPU load, memory usage, and free disk space on both hosts.

Example: Checking resource usage with the top command

top -b -n 1 | head -n 5

This command displays the first 5 lines of the top output, showing the total CPU load, memory usage, and other important metrics. The `-b` parameter runs top in batch mode, and `-n 1` specifies that only one iteration should be performed. This is useful for automation and scripting.

Example: Checking free disk space with the df command

df -h /var/lib/libvirt/images

This command shows disk space usage for the `/var/lib/libvirt/images` directory, where virtual machine images are typically stored. The `-h` option makes the output more readable by showing sizes in a human-readable format (e.g., GB, MB). Make sure the target host has enough free space to store the virtual machine disks.

Setting up Shared Storage (if necessary)

If you are using shared storage (e.g., NFS, iSCSI), make sure it is accessible and configured correctly on both hosts. Shared storage simplifies migration because the virtual machine continues to use the same disks, regardless of which host it is running on.

Example: Mounting an NFS share

mount -t nfs 192.168.1.10:/exports/vm_images /var/lib/libvirt/images

This command mounts an NFS share with the IP address 192.168.1.10 and the path `/exports/vm_images` to the `/var/lib/libvirt/images` directory on the local host. To make the mount permanent, add the appropriate entry to the `/etc/fstab` file.

Example: Adding an entry to /etc/fstab

echo "192.168.1.10:/exports/vm_images /var/lib/libvirt/images nfs defaults 0 0" >> /etc/fstab

This command adds a line to the `/etc/fstab` file that specifies how to mount the NFS share when the system boots. The `defaults` option uses the standard mount parameters, and `0 0` disables backup and file system checking.

Configuring Access Rights

Make sure that the user performing the migration has the necessary access rights on both hosts. This is usually the root user or a user who is a member of the libvirt group. Insufficient access rights can lead to errors when trying to connect to the host or manage virtual machines.

Example: Adding a user to the libvirt group

usermod -a -G libvirt your_username

This command adds the user `your_username` to the `libvirt` group. After executing the command, you need to log out and log back in for the changes to take effect.

Example: Checking group membership

groups your_username

This command shows the list of groups that the user `your_username` is a member of. Make sure that the `libvirt` group is in the list.

Configuring the Network for Migration

Proper network configuration is critical to a successful migration. It is recommended to use a dedicated network for migration traffic to avoid overloading the main network and to ensure stable and fast data transfer. In this section, we will look at setting up a dedicated network, configuring DNS, and configuring a firewall.

Setting up a Dedicated Network

Create a separate virtual network (e.g., VLAN) or physical network dedicated exclusively to migration traffic. Assign each host IP addresses on this network and make sure they can communicate with each other. Using a dedicated network improves migration security and performance.

Example: Configuring IP addresses on interfaces

# Host 1
ip addr add 192.168.10.1/24 dev eth1
ip link set eth1 up

# Host 2
ip addr add 192.168.10.2/24 dev eth1
ip link set eth1 up

These commands assign IP addresses 192.168.10.1 and 192.168.10.2 to the eth1 interface on hosts 1 and 2, respectively. Make sure the eth1 interface is connected to the dedicated migration network. The command `ip link set eth1 up` activates the interface.

Example: Checking communication between hosts

ping 192.168.10.2

This command checks the connection between host 1 and host 2 over the dedicated network. Make sure the pings are successful.

Configuring DNS

Make sure hostnames resolve correctly to the IP addresses of the dedicated network. This can be done by adding entries to the `/etc/hosts` file on both hosts or by configuring a DNS server. Correct name resolution simplifies the migration process and makes it more reliable.

Example: Adding entries to /etc/hosts

# Host 1 and Host 2
echo "192.168.10.1 host1.example.com host1" >> /etc/hosts
echo "192.168.10.2 host2.example.com host2" >> /etc/hosts

These commands add entries to the `/etc/hosts` file, linking the IP addresses of the dedicated network to the hostnames. Replace `host1.example.com` and `host2.example.com` with the fully qualified domain names of your hosts, and `host1` and `host2` with the short hostnames.

Example: Checking name resolution

ping host2

This command checks whether the hostname `host2` resolves correctly to the IP address 192.168.10.2. Make sure the pings are successful.

Configuring the Firewall

Configure the firewall on both hosts to allow traffic necessary for migration. Typically, these are the ports used by libvirt (e.g., TCP 16509). An incorrectly configured firewall can block migration traffic and lead to failures.

Example: Allowing libvirt traffic with firewalld

# Host 1 and Host 2
firewall-cmd --permanent --add-port=16509/tcp
firewall-cmd --reload

These commands add a rule to firewalld that allows TCP traffic on port 16509. The `—permanent` option makes the rule permanent, and `—reload` reloads the firewall to apply the changes.

Example: Checking firewall rules

firewall-cmd --list-all

This command displays a list of all rules configured in firewalld. Make sure there is a rule in the list that allows TCP traffic on port 16509.

Checking Host Compatibility

For a successful migration, it is crucial that the source and target hosts are compatible. This includes checking CPU compatibility, libvirt versions, and QEMU versions. Incompatibility can lead to migration failures or unstable virtual machine operation after migration. In this section, we will look at how to check compatibility using virsh and other tools.

Checking CPU Compatibility

Different CPU models may have different instruction sets, which can lead to migration issues. Use virsh capabilities to compare the CPU capabilities of the source and target hosts. If the CPUs are incompatible, you can use the --cpu-model option when migrating to specify a common CPU model supported by both hosts.

Example: Getting CPU information using virsh capabilities

virsh capabilities | grep -A 20 'cpu>'

This command displays CPU information using the `virsh capabilities` command and filters the output to show only the section starting with the `` tag. The `-A 20` parameter specifies that 20 lines should be displayed after the found line.

Example: Migrating with a specified CPU model

virsh migrate --live vm1 qemu+ssh://host2.example.com/system --cpu-model core2duo

This command performs a live migration of the `vm1` virtual machine to the `host2.example.com` host and specifies that the `core2duo` CPU model should be used. This option may be necessary if the CPUs of the source and target hosts are not fully compatible. Replace `core2duo` with a CPU model supported by both hosts.

Checking libvirt and QEMU Versions

Make sure the libvirt and QEMU versions on the source and target hosts are compatible. Using different versions can lead to virtual machine management issues and migration failures. It is recommended to use the same or similar versions of libvirt and QEMU on all hosts.

Example: Checking the libvirt version

virsh version

This command displays information about the libvirt version and the hypervisor used (e.g., QEMU). Check the libvirt version on both hosts and make sure they are compatible.

Example: Checking the QEMU version

qemu-system-x86_64 --version

This command displays QEMU version information. Check the QEMU version on both hosts and make sure they are compatible.

Checking Virtual Machine Configuration

Check the configuration of the virtual machine (e.g., number of vCPUs, amount of memory, network interfaces) to ensure that the target host supports it. Incompatibility in the configuration can lead to problems with starting the virtual machine after migration.

Example: Getting virtual machine information using virsh

virsh dominfo vm1

This command displays information about the `vm1` virtual machine, including ID, name, state, number of vCPUs, amount of memory, and other parameters. Compare this information with the resources of the target host.

Example: Getting the virtual machine’s XML configuration

virsh dumpxml vm1

This command displays the XML configuration of the `vm1` virtual machine. You can use this information for a more detailed analysis of the configuration and to check compatibility with the target host.

Performing Migration with virsh

virsh is a powerful command-line tool for managing virtual machines in a KVM environment. It provides extensive migration capabilities, including live migration, which allows you to move virtual machines between hosts without interrupting their operation. In this section, we will take a detailed look at how to perform migration with virsh, as well as look at various options and examples.

Basic Migration Commands

The main command for migrating virtual machines with virsh is virsh migrate. It takes various options that allow you to customize the migration process to suit your needs. Below are the main options:

• --live: Performs a live migration, that is, moves the virtual machine without interrupting its operation.
• --copy-storage: Copies the virtual machine’s storage to the target host.
• --persistent: Makes the virtual machine configuration persistent on the target host.
• --auto-converge: Enables automatic convergence to reduce downtime.
• --cpu-model: Specifies the CPU model to use on the target host (see previous section).

Example: Live migration with minimal options

virsh migrate --live vm1 qemu+ssh://host2.example.com/system

This command performs a live migration of the `vm1` virtual machine to the `host2.example.com` host. It uses the SSH protocol to connect to the target host. Make sure that the user executing the command has access rights on the target host.

Migration with Shared Storage

If the virtual machine uses shared storage, migration becomes easier because you do not need to copy the virtual machine disks. In this case, the migration command can be simplified.

Example: Live migration with shared storage

virsh migrate --live vm1 qemu+ssh://host2.example.com/system

This command performs a live migration of the `vm1` virtual machine to the `host2.example.com` host. Since shared storage is used, the `—copy-storage` option is not required.

Migration without Shared Storage

If the virtual machine does not use shared storage, you must copy the virtual machine disks to the target host. This can be done using the `—copy-storage` option or by other methods, such as rsync.

Example: Live migration with storage copying

virsh migrate --live --copy-storage vm1 qemu+ssh://host2.example.com/system

This command performs a live migration of the `vm1` virtual machine to the `host2.example.com` host and copies the virtual machine’s storage to the target host. This process can take some time, depending on the size of the virtual machine disks.

Example: Migration using rsync (alternative method)

# On the source host
rsync -avz /var/lib/libvirt/images/vm1.img host2.example.com:/var/lib/libvirt/images/

# After rsync completes, perform the migration
virsh migrate --live vm1 qemu+ssh://host2.example.com/system

This example shows how to copy the virtual machine’s `vm1.img` disk to the target host using rsync before performing the migration. The `-avz` option provides archiving, recursive copying, and data compression. After copying is complete, execute the virsh migrate command without the `—copy-storage` option.

Troubleshooting

Virtual machine migration, especially live migration, can encounter various problems. It is important to be able to diagnose and troubleshoot these problems to ensure successful virtual machine movement and avoid downtime. In this section, we will look at the most common problems and how to solve them.

Connection Issues

One of the most common problems is the inability to establish a connection between the source and target hosts. This may be due to network issues, a firewall, or DNS settings.

Example: Checking the connection with ping

ping host2.example.com

Make sure the hosts can communicate with each other over the network. If pings fail, check network settings, firewall, and DNS.

Example: Checking firewall rules

firewall-cmd --list-all

Make sure the firewall is not blocking traffic required for migration (e.g., port 16509 for libvirt). Add the necessary rules and reload the firewall.

Resource Issues

Lack of resources on the target host can cause the migration to fail. Make sure the target host has enough CPU, memory, and disk space to accommodate the virtual machine.

Example: Checking resource usage with top

top

Check CPU load and memory usage on the target host. If resources are heavily loaded, free them or move some of the load to another host.

Example: Checking free disk space with df

df -h

Make sure the target host has enough free space to store the virtual machine disks. If there is not enough space, increase the partition size or move the disks to another storage location.

CPU Compatibility Issues

CPU incompatibility between the source and target hosts can cause the migration to fail or the virtual machine to run unreliably after migration. Use the --cpu-model option when migrating to specify a common CPU model supported by both hosts.

Example: Migrating with a specified CPU model

virsh migrate --live vm1 qemu+ssh://host2.example.com/system --cpu-model core2duo

Specify a CPU model supported by both hosts. If you do not know which model to use, try host or kvm64.

Viewing Logs

If the migration fails, review the libvirt and QEMU logs for more information about the error. Logs can contain clues about the cause of the problem and help in resolving it.

Example: Viewing libvirt logs

tail -f /var/log/libvirt/libvirtd.log

This command displays the latest lines of the libvirt log file in real time. Look for error messages or warnings related to the migration.

Example: Viewing QEMU logs

tail -f /var/log/libvirt/qemu/vm1.log

This command displays the latest lines of the QEMU log file for the `vm1` virtual machine in real time. Look for error messages or warnings related to starting or migrating the virtual machine.