Unable to revert to VM snapshot after virtual machine stop/start: "Target CPU check full does not match source partial"

[phsm]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Virtual Machine snapshot

CLOUDSTACK VERSION

4.17.2

CONFIGURATION

OS / ENVIRONMENT

• KVM hypervisors
• Ubuntu 20.04
• Libvirt 6.0.0-0ubuntu8.16

SUMMARY

Error while reverting to VM snapshot after stop/start or migrate VM

STEPS TO REPRODUCE

1. Create/start a virtual machine.
2. Make a virtual machine snapshot with memory checkbox enabled
3. Stop/start or migrate the virtual machine to another host.
4. Try to revert the VM to the snapshot

EXPECTED RESULTS

The VM should be reverted to the snapshot

ACTUAL RESULTS

The following error occurs:
Invocation exception, caused by: 
com.cloud.utils.exception.CloudRuntimeException:
Revert VM: i-6309-123846-VM to snapshot: i-6309-123846-VM_VS_20230514183015 failed due to  
Revert to VM snapshot failed due to org.libvirt.LibvirtException: 
revert requires force: Target CPU check full does not match source partial

ADDITIONAL INFO

This error seems to come from libvirt, not cloudstack.
When making a VM snapshot, libvirt embeds the "metadata" of the virtual machine inside the snapshot. It is done to check the snapshot VM properties (cpu flags, amount of cores, devices models etc.) against the running VM properties before reverting to a snapshot. If they don't match then the snapshot can only be restored with --force flag as the integrity of the VM memory contents can not be guaranteed anymore.

However, it is odd that the issue happens even on a same node just after stop/start the virtual machine (nothing changes in between).

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kiranchavala]

@phsm could you check if the global setting "Kvm vmstoragesnapshot enabled" is enabled or not

[kiranchavala]

Please enable it and give it a try

[phsm]

Hello @kiranchavala

I tried setting this setting to true (even though the bug is related to virtual machine snapshot, not disk snapshot).

Here is what I tested:

1. Stopped the test virtual machine.
2. Changed the kvm.vmstoragesnapshot.enabled to True
3. Restarted cloudstack-management
4. Started the test virtual machine
5. Made a VM snapshot with "Snapshot memory" checkbox enabled.
6. Once the snapshot was made, stopped and started the test virtual machine.
7. Tried reverting to the snapshot
8. Got the "Target CPU check full does not match source partial" error.

I also tried to make a snapshot without "Snapshot memory" option enabled. It returns the following error: "Failed to create vm snapshot: KVM does not support the type of snapshot requested".

The description of the "kvm.vmstoragesnapshot.enabled" option says it requires qemu-guest-agent installed on the VM. I verified that it is working by issuing the guest-info command:

virsh # qemu-agent-command i-2-18926-VM '{"execute":"guest-info"}'
{"return":{"version":"6.2.0","supported_commands":[{"enabled":true,"name":"guest-ssh-remove-authorized-keys","success-response":true},{"enabled":true,"name":"guest-ssh-add-authorized-keys","success-response":true},{"enabled":true,"name":"guest-ssh-get-authorized-keys","success-response":true},{"enabled":false,"name":"guest-get-devices","success-response":true},{"enabled":true,"name":"guest-get-osinfo","success-response":true},{"enabled":true,"name":"guest-get-timezone","success-response":true},{"enabled":true,"name":"guest-get-users","success-response":true},{"enabled":true,"name":"guest-get-host-name","success-response":true},{"enabled":false,"name":"guest-exec","success-response":true},{"enabled":false,"name":"guest-exec-status","success-response":true},{"enabled":true,"name":"guest-get-memory-block-info","success-response":true},{"enabled":true,"name":"guest-set-memory-blocks","success-response":true},{"enabled":true,"name":"guest-get-memory-blocks","success-response":true},{"enabled":true,"name":"guest-set-user-password","success-response":true},{"enabled":true,"name":"guest-get-fsinfo","success-response":true},{"enabled":true,"name":"guest-get-disks","success-response":true},{"enabled":true,"name":"guest-set-vcpus","success-response":true},{"enabled":true,"name":"guest-get-vcpus","success-response":true},{"enabled":true,"name":"guest-network-get-interfaces","success-response":true},{"enabled":true,"name":"guest-suspend-hybrid","success-response":false},{"enabled":true,"name":"guest-suspend-ram","success-response":false},{"enabled":true,"name":"guest-suspend-disk","success-response":false},{"enabled":true,"name":"guest-fstrim","success-response":true},{"enabled":true,"name":"guest-fsfreeze-thaw","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze-list","success-response":true},{"enabled":true,"name":"guest-fsfreeze-freeze","success-response":true},{"enabled":true,"name":"guest-fsfreeze-status","success-response":true},{"enabled":false,"name":"guest-file-flush","success-response":true},{"enabled":false,"name":"guest-file-seek","success-response":true},{"enabled":false,"name":"guest-file-write","success-response":true},{"enabled":false,"name":"guest-file-read","success-response":true},{"enabled":false,"name":"guest-file-close","success-response":true},{"enabled":false,"name":"guest-file-open","success-response":true},{"enabled":true,"name":"guest-shutdown","success-response":false},{"enabled":true,"name":"guest-info","success-response":true},{"enabled":true,"name":"guest-set-time","success-response":true},{"enabled":true,"name":"guest-get-time","success-response":true},{"enabled":true,"name":"guest-ping","success-response":true},{"enabled":true,"name":"guest-sync","success-response":true},{"enabled":true,"name":"guest-sync-delimited","success-response":true}]}}

[weizhouapache]

added to 4.18.1.0 milestone

[weizhouapache]

@phsm
can you share the agent.log when you start the vm, create vm snapshot, and restore vm from snapshot?

[phsm]

@weizhouapache
Sure, here are the logs:
step1_create_and_start_vm.txt
step2_take_vm_snapshot.txt
step3_restore_from_snapshot.txt

Step1: Create a new VM and start it on node01
Step2: Make a VM snapshot with "VM memory" enabled
.. Then stop the VM and start it back. The hypervisor remains the same ..
Step3: Trying to revert to the snapshot.

[phsm]

It seems that this is a bug in Libvirt 6.x.x that is shipped with Ubuntu 20.04.

I tested it on Cloudstack 4.18 + Ubuntu 22.04 hypervisors (Libvirt 8.0.0), and the erroneous behavior isn't there anymore.

[weizhouapache]

It seems that this is a bug in Libvirt 6.x.x that is shipped with Ubuntu 20.04.

I tested it on Cloudstack 4.18 + Ubuntu 22.04 hypervisors (Libvirt 8.0.0), and the erroneous behavior isn't there anymore.

great, thanks for the update @phsm

can we close the issue ?