On my Xen host (systemd in Debian Buster, Xen 4.11), I usually start all guest VMs automatically. I added all my VM configs as link in
/etc/xen/auto/
to the underlying config like
/etc/xen/auto/proxy.cfg -> /etc/xen/proxy.cfg
/etc/xen/auto/vm04.matrix.cfg -> /etc/xen/vm04.matrix.cfg
...
and sometimes some guest VMs start automatically on reboot of the DOM0 host and sometimes some fail and throw these errors in syslog:
$ grep "Starting Xen domain" /var/log/syslog
May 12 08:38:36 dom0 xendomains: Starting Xen domain proxy (from /etc/xen/auto/proxy.cfg)...done.
May 12 08:38:36 dom0 xendomains: Starting Xen domain vm04.matrix.de (from /etc/xen/auto/vm04.matrix.cfg)...failed.
If they fail, the errors in \var\log\syslog
around the fail during boot are:
Starting Xen domain vm04.matrix (from /etc/xen/auto/vm04.matrix.cfg)...failed.
libxl: error: libxl_device.c:417:libxl__device_disk_set_backend: Disk vdev=xvda2 failed to stat: /dev/vg0/vm04.matrix-disk: No such file or directory
libxl: error: libxl_create.c:983:initiate_domain_create: Domain 4:Unable to set disk defaults for disk 0
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 4:Destruction of domain failed
---
Starting Xen domain proxy (from /etc/xen/auto/proxy.cfg)...failed.
libxl: error: libxl_device.c:417:libxl__device_disk_set_backend: Disk vdev=xvda2 failed to stat: /dev/vg0/proxy-disk: No such file or directory
libxl: error: libxl_create.c:983:initiate_domain_create: Domain 1:Unable to set disk defaults for disk 0
libxl: error: libxl_domain.c:1034:libxl__destroy_domid: Domain 1:Non-existant domain
libxl: error: libxl_domain.c:993:domain_destroy_callback: Domain 1:Unable to destroy guest
libxl: error: libxl_domain.c:920:domain_destroy_cb: Domain 1:Destruction of domain failed
The files /etc/xen/proxy.cfg
and /etc/xen/vm04.matrix.cfg
are identical apart from the mount-points.
If I start xen create /etc/xen/vm04.matrix.cfg
later manually it starts just fine, only not automatically during the boot of the host.
## debugging
To debug the problem, I added this in /etc/default/grub
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=4096M,max:4096M loglvl=all guest_loglvl=all"
and called update-grub
, but after a reboot the error messages in syslog are the same. I cannot find anything suspicious in dmesg
either. No mentions of any of the guest hostnames. Only this shows, the problem:
# dmesg|grep dev|tail
[ 11.348931] xen:xen_evtchn: Event-channel device installed
[ 22.006020] device vif3.0 entered promiscuous mode
[ 25.362878] device vif5.0 entered promiscuous mode
[ 28.636912] device vif6.0 entered promiscuous mode
[ 1004.501654] device vif7.0 entered promiscuous mode
...
You only see, that the first three VMs Network-interfaces did start during boot this time and 1000s later I started the next VM manually on the console.
Xen loads just fine:
$ systemctl --type=service | grep -i xen
xen.service loaded active running LSB: Xen daemons
xendomains.service loaded active running LSB: Start/stop secondary xen domains
I think the best hint to the problem is that the volume is not there during boot:
/dev/vg0/proxy-disk: No such file or directory
maybe xen starts too early in the boot process?
systemctl cat xen.service|uniq
# /run/systemd/generator.late/xen.service
# Automatically generated by systemd-sysv-generator
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/xen
Description=LSB: Xen daemons
Before=multi-user.target
Before=graphical.target
After=remote-fs.target
[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/xen start
ExecStop=/etc/init.d/xen stop
systemctl cat xendomains.service|uniq
# /run/systemd/generator.late/xendomains.service
# Automatically generated by systemd-sysv-generator
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/xendomains
Description=LSB: Start/stop secondary xen domains
Before=multi-user.target
Before=graphical.target
Before=corosync.service
Before=heartbeat.service
Before=libvirtd.service
After=network-online.target
After=remote-fs.target
After=xen.service
After=drbd.service
After=iscsi.service
After=openvswitch-switch.service
After=nfs-kernel-server.service
Wants=network-online.target
[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/xendomains start
ExecStop=/etc/init.d/xendomains stop
ExecReload=/etc/init.d/xendomains reload
How can I further debug this?
Asked by rubo77
(30435 rep)
May 12, 2020, 01:06 PM
Last activity: Jun 12, 2024, 03:23 PM
Last activity: Jun 12, 2024, 03:23 PM