How to debug a loop in systemd-shutdown / initrd ? (Arch Linux)
1
vote
0
answers
654
views
I've recently installed Arch Linux to a virtual disk, under a custom configuration for root on ZFS. This was in using some scripting I've put together for managing a multiboot configuration, with separate boot and root pools, each to be used both for Arch and FreeBSD. Presently, the ZFS pool configuration was created with FreeBSD under a VM. The Arch Linux installation was then created from another VM, where Arch was installed on an Ext4 partition. With Arch installed on the boot/root ZFS configuration and after some configuration for Grub, the Arch installation on ZFS now boots successfully.
Presently, the virtual disk with the ZFS pools on it is running under a separate virtual machine. Arch Linux is the only OS installed on that machine's virtual disk, at this time. It boots successfully now, with a corresponding Grub2 configuration. However, I'm noticing a loop during shutdown.
[ 1561.324253] systemd-shutdown[1] : Sending SIGKILL to remaining processes...
[ 1561.336930] systemd-shutdown[1] : Unmounting file systems.
[ 1561.341650] : Remounting '/var' read-only in with options 'noxattr,noacl'.
[ 1561.783555] : Unmounting '/var'.
[ 1561.790194] : Remounting '/usr' read-only in with options 'noxattr,noacl'.
[ 1561.796853] : Remounting '/' read-only in with options 'noxattr,noacl'.
[ 1561.803420] systemd-shutdown[1] : All filesystems unmounted.
[ 1561.809423] systemd-shutdown[1] : Deactivating swaps.
[ 1561.813672] systemd-shutdown[1] : All swaps deactivated.
[ 1561.818688] systemd-shutdown[1] : Detaching loop devices.
[ 1561.824354] systemd-shutdown[1] : All loop devices detached.
[ 1561.829085] systemd-shutdown[1] : Stopping MD devices.
[ 1561.833346] systemd-shutdown[1] : All MD devices stopped.
[ 1561.837982] systemd-shutdown[1] : Detaching DM devices.
[ 1561.842299] systemd-shutdown[1] : All DM devices detached.
[ 1561.847501] systemd-shutdown[1] : All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 1561.856852] systemd-shutdown[1] : Successfully changed into root pivot.
[ 1561.863478] systemd-shutdown[1] : Returning to initrd...
[ 1562.306958] shutdown[1] : Syncing filesystems and block devices.
[ 1562.311377] shutdown[1] : Sending SIGTERM to remaining processes...
[ 1562.316903] shutdown[1] : Sending SIGKILL to remaining processes...
[ 1562.329374] shutdown[1] : Unmounting file systems.
[ 1562.334319] sd-remou: Remounting '/' read-only in with options 'noxattr,noacl'.
[ 1562.346902] sd-umoun: Unmounting '/oldroot/sys/kernel/config'.
[ 1562.354595] sd-umoun: Unmounting '/oldroot/sys/fs/fuse/connections'.
[ 1562.361525] sd-umoun: Unmounting '/oldroot/sys/kernel/tracing'.
[ 1562.368232] sd-umoun: Unmounting '/oldroot/sys/kernel/debug'.
[ 1562.373715] sd-umoun: Unmounting '/oldroot/dev/mqueue'.
[ 1562.380105] sd-umoun: Unmounting '/oldroot/dev/hugepages'.
[ 1562.386843] sd-umoun: Unmounting '/oldroot/sys/fs/bpf'.
[ 1562.391984] sd-umoun: Unmounting '/oldroot/sys/fs/pstore'.
[ 1562.397417] sd-umoun: Unmounting '/oldroot/sys/fs/cgroup'.
[ 1562.404234] audit: type=1334 audit(1635363440.667:81): prog-id=8 op=UNLOAD
[ 1562.410100] sd-umoun: Unmounting '/oldroot/dev/pts'.
[ 1562.414579] audit: type=1334 audit(1635363440.667:82): prog-id=6 op=UNLOAD
[ 1562.420101] sd-umoun: Unmounting '/oldroot/dev/shm'.
[ 1562.426852] sd-umoun: Unmounting '/oldroot/sys/kernel/security'.
[ 1562.432165] sd-umoun: Unmounting '/oldroot/usr'.
[ 1562.443869] sd-remou: Remounting '/oldroot/run' read-only in with options 'noxattr,noacl'.
[ 1562.456905] sd-umoun: Unmounting '/oldroot/run'.
[ 1562.461323] sd-umoun: Unmounting '/oldroot'.
[ 1562.466800] sd-umoun: Failed to unmount /oldroot: Device or resource busy
[ 1562.473458] sd-umoun: Unmounting '/oldroot/dev'.
[ 1562.478395] sd-umoun: Unmounting '/oldroot/sys'.
[ 1562.483288] sd-umoun: Unmounting '/oldroot/proc'.
[ 1562.488082] sd-umoun: Unmounting '/oldroot'.
[ 1562.493279] shutdown[1] : All filesystems unmounted.
[ 1562.498851] shutdown[1] : Deactivating swaps.
[ 1562.502456] shutdown[1] : All swaps deactivated.
[ 1562.506839] shutdown[1] : Detaching loop devices.
[ 1562.511505] shutdown[1] : All loop devices detached.
[ 1562.515404] shutdown[1] : Stopping MD devices.
[ 1562.520083] shutdown[1] : All MD devices stopped.
[ 1562.523666] shutdown[1] : Detaching DM devices.
[ 1562.527617] shutdown[1] : All DM devices detached.
[ 1562.531822] shutdown[1] : All filesystems, swaps, loop devices, MD devices and DM devices detached.
[ 1562.540145] shutdown[1] : Successfully changed into root pivot.
[ 1562.544940] shutdown[1] : Returning to initrd...
I've only encountered this with the Arch Linux installation on ZFS. Using the same systemd, mkinitcpio, linux, and zfs-linux versions under another VM installation, with Arch on an Ext4 partition there, there's no such loop occurring during shutdown in that machine. May it be due to the root on ZFS configuration?
I'm at a loss for trying to figure out how to debug this loop any further.
To my best guess, this may be related to some scripting for systemd-shutdown on mkinticpio, such that was introduced in 2013 (Bächler2013 ).
What appears to be the main code in the mkinitcpio-generate-shutdown-ramfs
service, introduced around that time - an excerpt of /usr/lib/systemd/system/mkinitcpio-generate-shutdown-ramfs.service
[Service]
Type=oneshot
# /tmp could be umounted at this point
# use /run as temporary directory
Environment=TMPDIR=/run
ExecStart=/usr/bin/mkinitcpio -A sd-shutdown -k none -c /dev/null -d /run/initramfs
Furthermore, the contents of the file /usr/lib/initcpio/install/sd-shutdown
#!/bin/bash
build() {
add_binary /usr/lib/systemd/systemd-shutdown /shutdown
if type -P kexec >/dev/null; then
add_binary kexec
fi
}
help() {
cat <
I'm not certain if that's actually making it into the shutdown process, on the VM - the system might not meet the stated preconditions for that much.
I've tried masking the mkinitcpio-generate-shutdown-ramfs
service under systemd, then running mkinitcpio -P
again, rebooting ... and it still goes into that loop during system shutdown.
The system is still usable under the VM installation - at least, up to the point of that event in the shutdown, when the machine begins to loop indefinitely. Ideally I'd like to be able to work this knot out of the configuration, before installing anything beside Arch or putting this onto a machine's internal hard disk.
I'm not certain what may be resulting in the message, Failed to unmount /oldroot: Device or resource busy
. It may be something related to the root on ZFS installation? But that's just a guess, really.
To my own perspective, the systemd-shutdown process - is it running under initrd, from what the console output says? I believe it seems nontrivial to debug, at this point.
I'm certain I could continue towards using Arch for keeping the Grub configuration up to date, after installing FreeBSD beside Arch, in the ZFS pool configuration under the VM's virtual disk. If it's possible to fix this issue with the shutdown loop under Arch, however, I'd like to be able to maintain the Arch installation as a working desktop system - in the VM installation, or finally on the PC's hard disk.
While I'm at a loss for trying to figure out how to debug this loop any further, for that loop during shutdown, maybe there's a patch that I just haven't found yet?
**Update**
I've found source code for systemd-shutdown, at the version in use on Arch Linux. I think it looks like some fairly straightforward code. There may not be a lot there about the 'oldroot' mount point, or how it could become unmountable under systemd-shutdown.
It doesn't appear to be due to logging. I've set Storage=volatile
in /etc/systemd/journald.conf
and created a oneshot systemd service that runs /usr/bin/journalctl --flush
after boot. The /var/
filesystem gets successfully umounted now, and I don't believe I'm loosing all of the log data across reboots.
The loop is still occurring during the shutdown, however, along with the message Failed to unmount /oldroot...
from systemd-shutdown. Is there something hanging it up by side effect?
Asked by Sean Champ
(266 rep)
Nov 2, 2021, 02:40 AM
Last activity: Nov 2, 2021, 03:41 PM
Last activity: Nov 2, 2021, 03:41 PM