Virtual Router VM fails to boot with kernel panic when overprovisioinig is > 1

[MilanHofmann]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Virtual Router VM / service offering / shared storage

CLOUDSTACK VERSION

4.18

CONFIGURATION

Advanced Zone with NFS primary Storage.

OS / ENVIRONMENT

Ubuntu Focal Fossa

SUMMARY

When overprovisioning factors are > 1, the virtual router vm for a new zone is not able to boot and ends up in a kernel panic while trying to mount the root volume.
By default a local storage service offering is picked for the router.
If an id of a service offering that uses shared storage is specified in the global settings under router.service.offering it is ignored.
Somehow the Volumes attached to the working system and router vms (when overprovisioning == 1) are not accessible via the UI and lead to 404.
Taking a look at the db entry they look like:

                    id: 26
            account_id: 1
             domain_id: 1
               pool_id: 4
          last_pool_id: NULL
           instance_id: 21
             device_id: 0
                  name: ROOT-21
                  uuid: 29537b97-e6d1-484b-9e06-92cd905ead8e
                  size: 5242880000
                folder: NULL
                  path: 29537b97-e6d1-484b-9e06-92cd905ead8e
                pod_id: NULL
        data_center_id: 4
            iscsi_name: NULL
               host_ip: NULL
           volume_type: ROOT
             pool_type: NULL
      disk_offering_id: 12
           template_id: 3
           first_snapshot_backup_uuid: NULL
           recreatable: 1
               created: 2023-04-10 11:51:38
              attached: NULL
               updated: 2023-04-10 11:51:40
               removed: NULL
                 state: Ready
            chain_info: NULL
          update_count: 2
             disk_type: NULL
vm_snapshot_chain_size: NULL
                iso_id: NULL
        display_volume: 1
                format: QCOW2
              min_iops: NULL
              max_iops: NULL
         hv_ss_reserve: NULL
     provisioning_type: thin
         external_uuid: NULL
         passphrase_id: NULL
        encrypt_format: NULL

The disk offering for the volume is type "shared".

In addition there is a second primary storage that was setup by accident, always comes up again (after disabling), but has no visible resources attached to it.

STEPS TO REPRODUCE

Set overprovisioning to > 1 and create a new VPC.

EXPECTED RESULTS

The virtual router vm should use the shared storage offering and boot.

ACTUAL RESULTS

It ends up ina kernel panic mounting the root volume.

[MilanHofmann]

I have no re-setup everything from scratch, the system offerings are beeing picked up.
But still the routers run out of memory when booting if overprovisioning > 1.
Is there anybody that can help me with this issue?

[rajujith]

@MilanHofmann what is the hypervisor type and version you are using? please review this thread https://lists.apache.org/thread/77rgh7qqk0ro90mhgbw0kr6jmsc785wk

[MilanHofmann]

@rajujith Thank you very much for the hint!
I am using KVM with Cloudstack 4.18, so this thread makes sense.
But how can I enable ballooning via cloudstack?
As far as I understand memory ballooning is a per-guest setting, so I'd have to make cloudstack make use of these options anytime a new vm is created, right?

[weizhouapache]

@MilanHofmann
what's the memory overprovision factor in global setting and cluster setting ?
it is suggested to set to 1 on kvm environments.

if you want to enable VM dynamic scaling , please refer to https://www.shapeblue.com/granular-control-of-dynamic-scaling-of-vms-cpu-ram/

[rajujith]

@MilanHofmann you can go ahead with Wei's suggestion without using memory overcommit. I would like the below to understand what is causing the issue:

1. OS version for KVM
2. memory overcommit value at the cluster or VM level.
3. Output of ‘virsh dumpxml vm-name’ on KVM

[rajujith]

Testing results :

Offering 512MB, with mem overcommit :1

[root@ref-trl-4866-k-m7-jithin-raju-kvm2 ~]# virsh dumpxml i-2-3-VM

<domain type='kvm' id='3'>
  <name>i-2-3-VM</name>
  <uuid>32885d38-c0f9-4878-a722-2a32061ab943</uuid>
  <description>CentOS 5.5 (64-bit)</description>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>

[root@vm1 ~]# grep MemTotal /proc/meminfo
MemTotal: 510584 kB

After changing memory overcommit to 2 on cluster:

<domain type='kvm' id='4'>
  <name>i-2-3-VM</name>
  <uuid>32885d38-c0f9-4878-a722-2a32061ab943</uuid>
  <description>CentOS 5.5 (64-bit)</description>
  <memory unit='KiB'>262144</memory>
  <currentMemory unit='KiB'>262144</currentMemory>

[root@vm1 ~]# grep MemTotal /proc/meminfo
MemTotal: 252416 kB

Ballooning config:

<memballoon model='virtio'>
     <alias name='balloon0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
   </memballoon>

Same is applicable for VRs,systemVMs

[root@ref-trl-4866-k-m7-jithin-raju-kvm2 ~]# virsh dominfo 7
setlocale: No such file or directory
Id: 7
Name: s-1-VM
UUID: 49d1f0c2-6890-4bbb-a281-45e01850db25
OS Type: hvm
State: running
CPU(s): 1
CPU time: 25.3s
Max memory: 262144 KiB
Used memory: 262144 KiB
Persistent: no
Autostart: disable
Managed save: no
Security model: none
Security DOI: 0

With memory overprovisioning set to 1 with the default offering of VR

[root@ref-trl-4866-k-m7-jithin-raju-kvm1 ~]# virsh dominfo r-4-VM

Id: 1
Name: r-4-VM
UUID: 73787768-76bf-4b4e-a381-f0ed6b6bedf9
OS Type: hvm
State: running
CPU(s): 1
CPU time: 343.6s
Max memory: 262144 KiB
Used memory: 262144 KiB
Persistent: no
Autostart: disable
Managed save: no
Security model: none
Security DOI: 0

After changing the memory over provisioning to 2

[root@ref-trl-4866-k-m7-jithin-raju-kvm1 ~]# virsh dominfo r-4-VM
setlocale: No such file or directory
Id: 5
Name: r-4-VM
UUID: 73787768-76bf-4b4e-a381-f0ed6b6bedf9
OS Type: hvm
State: running
CPU(s): 1
CPU time: 136.0s
Max memory: 131072 KiB
Used memory: 131072 KiB
Persistent: no
Autostart: disable
Managed save: no
Security model: none
Security DOI: 0

With the above Memory the VR can't start:

If we use autodeflate=on in the ballooning configuration without reducing the max memory this should work as expected. i.e set Max memory to offering value, current memory to offering value/ memory overcommit. This should guarantee minimum Memory and the VM can use the Maximum memory if available and the Maximum memory will be accessible to the guest.

[1] https://www.linux-kvm.org/page/Projects/auto-ballooning

[2] https://www.humblec.com/memory-ballooning-and-virtio_balloon-driver-in-qemu-kvm/

[rajujith]

I would propose this solution:
Retain the 'memory' to the offering RAM value, set the currentMemory to offering RAM value/ memory over provisioning value. Also set the autodeflate='on' for the balloon driver. With this the guest will get the RAM specified in the offering.

<memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>524288</currentMemory>

  <memballoon model='virtio' autodeflate='on'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>

[1] https://libvirt.org/formatdomain.html#memory-balloon-device