GRUB not booting any more after system update: How to set the root variable the correct way?
2
votes
0
answers
38
views
Today I ran an upgrade on one of my debian 11 VMs to get the latest security updates. Obviously, the GRUB system has been broken by the update. The VM uses EFI, so there is a sort of "chainloading". First
EFI\debian\grub.cfg
is executed, which is a minimal configuration file that directs GRUB to the main configuration file /boot/grub/grub.cfg
.
The main configuration file lets GRUB search for the wrong UUID when trying to determine the root
environment variable. This prevents the system from booting.
These are my file system UUIDs:
root@morn ~ # blkid
/dev/vda2: UUID="4963-B5C0" BLOCK_SIZE="4096" TYPE="vfat" PARTUUID="40d4aada-c48d-446d-87e0-8a3ca2514eaf"
/dev/vda1: UUID="7c91164d-298d-4ef8-9823-df48a13e5325" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea35937f-4329-4c09-a674-70b551e654d9"
As we can see, /dev/vda2
is the EFI partition, while /dev/vda1
is the partition the system should boot from.
This is the respective snippet from my /boot/grub/grub.cfg
:
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-4963-B5C0' {
load_video
insmod gzio
if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=root 4963-B5C0
echo 'Loading Linux 6.1.0-37-amd64 ...'
linux /boot/vmlinuz-6.1.0-37-amd64 root=UUID=7c91164d-298d-4ef8-9823-df48a13e5325 ro ipv6.disable=1 quiet
echo 'Loading initial ramdisk ...'
initrd /boot/initrd.img-6.1.0-37-amd64
}
I have generated that grub.cfg
the recommended way via update-grub
from within the VM while it was running. Obviously, update-grub
uses the wrong UUID when generating the search ...
line. It uses the UUID of the EFI file system instead of the UUID of the linux root file system. Consequently, GRUB cannot boot because there is no kernel and no appropriate directory structure on the EFI partition.
As mentioned above, this happens since I have installed updates for Debian 11 in that VM. I'd like to emphasize that this was not a version upgrade. The VM was on Debian 11 since quite a while, and GRUB and its associated tools always worked flawlessly until I had applied the missing updates.
Of course, I now could simply edit /boot/grub/grub.cfg
every time I have run update-grub
, replacing the search ...
line by something like set root=(hd0,gpt1)
. But this approach is not recommended, and it is error prone. On the other hand, changing the custom scripts in /etc/grub.d
doesn't seem reasonable as well.
Does anybody know another way to tell update-grub
the correct UUID for that search ...
stanza?
If memory serves me correctly, we can disable the UUID search using "official" methods, but that seems a bad idea, and I really would like to learn how to tell GRUB the correct UUID and make it behave as before the upgrade.
Another solution would be to not care about the root
variable at all and simply prepend (hd0,gpt1)
before the linux
and the initrd
path. But I don't know a reasonable (safe for upgrades) way to do that either.
**EDIT / UPDATE 2025-08-07 #1**
In the meantime, I have researched further and have found something that is very suspicious and nearly surely causes the problem. Please consider the following snippet from the terminal in the VM in question:
root@morn /etc/grub.d # blkid
/dev/vda2: UUID="4963-B5C0" BLOCK_SIZE="4096" TYPE="vfat" PARTUUID="40d4aada-c48d-446d-87e0-8a3ca2514eaf"
/dev/vda1: UUID="7c91164d-298d-4ef8-9823-df48a13e5325" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea35937f-4329-4c09-a674-70b551e654d9"
root@morn /etc/grub.d # fdisk -l /dev/vda
Disk /dev/vda: 112 GiB, 120259084288 bytes, 29360128 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8C7D0CC5-3D22-490E-A1CC-92DA49B5D125
Device Start End Sectors Size Type
/dev/vda1 131072 29360122 29229051 111.5G Linux filesystem
/dev/vda2 16384 131071 114688 448M EFI System
Partition table entries are not in disk order.
root@morn /etc/grub.d # mount |grep /dev/vda
/dev/vda1 on / type ext4 (rw,relatime,quota,usrquota,grpquota,errors=remount-ro)
So far, we once again see without any doubt that the root file system ('/') is mounted on vda1
and that it is an ext4
file system with UUID 7c91164d-298d-4ef8-9823-df48a13e5325
. Furthermore, the two file systems on vda
definitely have different UUIDs.
And now GRUB seems to have a massive bug:
root@morn /etc/grub.d # grub-probe -d /dev/vda1; grub-probe -d /dev/vda2
fat
fat
root@morn /etc/grub.d # grub-probe -t fs_uuid -d /dev/vda1; grub-probe -t fs_uuid -d /dev/vda2
4963-B5C0
4963-B5C0
root@morn /etc/grub.d # grub-probe /; grub-probe -t fs_uuid /
fat
4963-B5C0
So GRUB obviously has come to the conclusion that both file systems (on vda1
and vda2
, respectively) are FAT
, that they both have the same UUID (4963-B5C0
), and that /
is mounted on a FAT
file system with that UUID.
Of course, this is complete nonsense and clearly contradicts the output of blkid
, mount
and fdisk
.
Any ideas?
**EDIT / UPDATE 2025-08-08 #1**
I'd like to draw everybody's attention to something that my be easily missed when reading the post:
As fdisk -l /dev/vda
shows, the disk's physical sector size as well the disk's logical sector size are 4096
in that VM. As far as I can tell, this is a quite unusual scenario. We had our reasons to set it up that way, though, and we don't want to change it.
Perhaps grub-probe
is not able to deal with logical sector sizes other than 512
. Can somebody confirm this?
Asked by Binarus
(3901 rep)
Aug 6, 2025, 06:26 PM
Last activity: Aug 8, 2025, 06:53 AM
Last activity: Aug 8, 2025, 06:53 AM