Ytdyklly

I have a VM on a host and want to create another VM within the existing VM (nested).

• Host: opensuse Tumbleweed, kernel 4.19.7, 96 CPUS Intel-Xeon, 256GB RAM (nested virt enabled in kvm)


• L1 VM: opensuse Leap, kernel 4.12.14, 4 vCPUS, 8 GB RAM


• L2 VM: opensuse Leap, 1 vCPU, 1GB RAM

Virtalization on L1 VM is apparently enabled:

$ virt-host-validate 

  QEMU: Checking for hardware virtualization                                 : PASS

  QEMU: Checking if device /dev/kvm exists                                   : PASS

  QEMU: Checking if device /dev/kvm is accessible                            : PASS

  QEMU: Checking if device /dev/vhost-net exists                             : PASS

  QEMU: Checking if device /dev/net/tun exists                               : PASS

  QEMU: Checking for cgroup 'cpu' controller support                         : PASS

  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS

  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS

  QEMU: Checking for cgroup 'memory' controller support                      : PASS

  QEMU: Checking for cgroup 'devices' controller support                     : PASS

  QEMU: Checking for cgroup 'blkio' controller support                       : PASS

  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)

   LXC: Checking for Linux >= 2.6.26                                         : PASS

   LXC: Checking for namespace ipc                                           : PASS

   LXC: Checking for namespace mnt                                           : PASS

   LXC: Checking for namespace pid                                           : PASS

   LXC: Checking for namespace uts                                           : PASS

   LXC: Checking for namespace net                                           : PASS

   LXC: Checking for namespace user                                          : PASS

   LXC: Checking for cgroup 'cpu' controller support                         : PASS

   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS

   LXC: Checking for cgroup 'cpuset' controller support                      : PASS

   LXC: Checking for cgroup 'memory' controller support                      : PASS

   LXC: Checking for cgroup 'devices' controller support                     : PASS

   LXC: Checking for cgroup 'freezer' controller support                     : PASS

   LXC: Checking for cgroup 'blkio' controller support                       : PASS

   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

And lscpu shows vmx flag:

 $ lscpu 

    Architecture:        x86_64

    CPU op-mode(s):      32-bit, 64-bit

    Byte Order:          Little Endian

    CPU(s):              4

    On-line CPU(s) list: 0-3

    Thread(s) per core:  1

    Core(s) per socket:  1

    Socket(s):           4

    NUMA node(s):        1

    Vendor ID:           GenuineIntel

    CPU family:          6

    Model:               79

    Model name:          Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz

    Stepping:            1

    CPU MHz:             2194.710

    BogoMIPS:            4389.42

    Virtualization:      VT-x

    Hypervisor vendor:   KVM

    Virtualization type: full

    L1d cache:           32K

    L1i cache:           32K

    L2 cache:            4096K

    L3 cache:            16384K

    NUMA node0 CPU(s):   0-3

    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch topoext cpuid_fault invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat umip

After creating a VM, it goes to paused mode and can't be resumed.

$ virsh list --all

 Id   Name   State

---------------------

 3    myvm   paused



$ virsh resume myvm

error: Failed to resume domain myvm

error: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

This is the qemu log showing some problem with kvm, hardware error 0x7.

$ cat /var/log/libvirt/qemu/myvm.log 

2018-12-19 15:34:27.814+0000: starting up libvirt version: 4.10.0, qemu version: 3.0.95openSUSE Leap 15.1, kernel: 4.12.14-lp151.19-default, hostname: susetest

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name guest=myvm,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-myvm/master-key.aes -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ff9104ff-4992-4747-bbd4-33c8517a9570 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=24,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/var/lib/libvirt/images/mydisk.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=be:a6:22:ed:1a:7b,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charchannel0 -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev pty,id=charconsole1 -device virtconsole,chardev=charconsole1,id=console1 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

char device redirected to /dev/pts/1 (label charserial0)

char device redirected to /dev/pts/2 (label charchannel0)

char device redirected to /dev/pts/3 (label charconsole1)

KVM: entry failed, hardware error 0x7

Same error, but starting a random live CD VM:

$ /usr/bin/qemu-system-x86_64 -vnc 0:5994 -enable-kvm -cdrom Fedora-17-x86_64-Live-Desktop.iso



KVM: entry failed, hardware error 0x7

EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000663

ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000

EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0

ES =0000 00000000 0000ffff 00009300

CS =f000 ffff0000 0000ffff 00009b00

SS =0000 00000000 0000ffff 00009300

DS =0000 00000000 0000ffff 00009300

FS =0000 00000000 0000ffff 00009300

GS =0000 00000000 0000ffff 00009300

LDT=0000 00000000 0000ffff 00008200

TR =0000 00000000 0000ffff 00008b00

GDT=     00000000 0000ffff

IDT=     00000000 0000ffff

CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000

DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 

DR6=00000000ffff0ff0 DR7=0000000000000400

EFER=0000000000000000

Code=06 66 05 00 00 01 00 8e c1 26 66 a3 14 f0 66 5b 66 5e 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00