2022-01-04T22:39:00

KVM shutdown hangs, process does not terminate after shutdown appears done

Struggled some time with kvm/qemu-system-x86_64 guest machines that did not terminate after "System halted" messages.

If I had to figure out how to get to the console output, disabling graphical boot output, to see what is going on. In Debian it's as easy as setting these variables in /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset"
GRUB_TERMINAL=console

Afterwards it was possible to start kvm quest machine with -curses option to see full shut-down log output in text mode:

systemd-journald[194]: Received SIGTERM from PID 1 (systemd-shutdow).
systemd-shutdown[1]: Sending SIGKILL to remaining processes...
systemd-shutdown[1]: Unmounting file systems.
[659]: Remounting '/' read-only in with options 'errors=remount-ro'.
EXT4-fs (sda2): re-mounted. Opts: errors=remount-ro
systemd-shutdown[1]: All filesystems unmounted.
systemd-shutdown[1]: Deactivating swaps.
systemd-shutdown[1]: All swaps deactivated.
systemd-shutdown[1]: Detaching loop devices.
systemd-shutdown[1]: All loop devices detached.
systemd-shutdown[1]: Stopping MD devices.
systemd-shutdown[1]: All MD devices stopped.
systemd-shutdown[1]: Detaching DM devices.
systemd-shutdown[1]: All DM devices detached.
systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
systemd-shutdown[1]: Syncing filesystems and block devices.
systemd-shutdown[1]: Halting system.
kvm: exiting hardware virtualization
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
reboot: System halted

Well and the kvm process just stopped there, never terminating. Searching around on web did not help much also there are couple of people asking similar questions mostly related to real hardware. At some point after nearly giving up, I've came across how to shutdown virtual machine remotely using ssh. Nothing special, but there it was, something else than halt command I was doing all the time - I've tried with shutdown -h now and poweroff commands and suddenly the guest machine processed also terminated:

systemd-shutdown[1]: Syncing filesystems and block devices.
systemd-shutdown[1]: Powering off.
kvm: exiting hardware virtualization
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
ACPI: Preparing to enter system sleep state S5
reboot: Power down

Well of course, right? :) All of these commands are symlinks:

/sbin/halt -> /bin/systemctl
/sbin/poweroff -> /bin/systemctl
/usr/sbin/shutdown -> /bin/systemctl

and all are documented in man systemctl. For halt it says:

Note that this operation will simply halt the OS kernel after shutting down, leaving the hardware powered on. Use systemctl poweroff for powering off the system (see below).

So it was doing exactly what it should have bin doing. Yes RTFM, but first figure out which one!