Sample Header Ad - 728x90

GRUB not booting any more after system update: How to set the root variable the correct way?

2 votes
0 answers
38 views
Today I ran an upgrade on one of my debian 11 VMs to get the latest security updates. Obviously, the GRUB system has been broken by the update. The VM uses EFI, so there is a sort of "chainloading". First EFI\debian\grub.cfg is executed, which is a minimal configuration file that directs GRUB to the main configuration file /boot/grub/grub.cfg. The main configuration file lets GRUB search for the wrong UUID when trying to determine the root environment variable. This prevents the system from booting. These are my file system UUIDs:
root@morn ~ # blkid
/dev/vda2: UUID="4963-B5C0" BLOCK_SIZE="4096" TYPE="vfat" PARTUUID="40d4aada-c48d-446d-87e0-8a3ca2514eaf"
/dev/vda1: UUID="7c91164d-298d-4ef8-9823-df48a13e5325" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea35937f-4329-4c09-a674-70b551e654d9"
As we can see, /dev/vda2 is the EFI partition, while /dev/vda1 is the partition the system should boot from. This is the respective snippet from my /boot/grub/grub.cfg:
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-4963-B5C0' {
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod fat
        search --no-floppy --fs-uuid --set=root 4963-B5C0
        echo    'Loading Linux 6.1.0-37-amd64 ...'
        linux   /boot/vmlinuz-6.1.0-37-amd64 root=UUID=7c91164d-298d-4ef8-9823-df48a13e5325 ro ipv6.disable=1 quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-6.1.0-37-amd64
}
I have generated that grub.cfg the recommended way via update-grub from within the VM while it was running. Obviously, update-grub uses the wrong UUID when generating the search ... line. It uses the UUID of the EFI file system instead of the UUID of the linux root file system. Consequently, GRUB cannot boot because there is no kernel and no appropriate directory structure on the EFI partition. As mentioned above, this happens since I have installed updates for Debian 11 in that VM. I'd like to emphasize that this was not a version upgrade. The VM was on Debian 11 since quite a while, and GRUB and its associated tools always worked flawlessly until I had applied the missing updates. Of course, I now could simply edit /boot/grub/grub.cfg every time I have run update-grub, replacing the search ... line by something like set root=(hd0,gpt1). But this approach is not recommended, and it is error prone. On the other hand, changing the custom scripts in /etc/grub.d doesn't seem reasonable as well. Does anybody know another way to tell update-grub the correct UUID for that search ... stanza? If memory serves me correctly, we can disable the UUID search using "official" methods, but that seems a bad idea, and I really would like to learn how to tell GRUB the correct UUID and make it behave as before the upgrade. Another solution would be to not care about the root variable at all and simply prepend (hd0,gpt1) before the linux and the initrd path. But I don't know a reasonable (safe for upgrades) way to do that either. **EDIT / UPDATE 2025-08-07 #1** In the meantime, I have researched further and have found something that is very suspicious and nearly surely causes the problem. Please consider the following snippet from the terminal in the VM in question:
root@morn /etc/grub.d # blkid
/dev/vda2: UUID="4963-B5C0" BLOCK_SIZE="4096" TYPE="vfat" PARTUUID="40d4aada-c48d-446d-87e0-8a3ca2514eaf"
/dev/vda1: UUID="7c91164d-298d-4ef8-9823-df48a13e5325" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea35937f-4329-4c09-a674-70b551e654d9"

root@morn /etc/grub.d # fdisk -l /dev/vda
Disk /dev/vda: 112 GiB, 120259084288 bytes, 29360128 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8C7D0CC5-3D22-490E-A1CC-92DA49B5D125

Device      Start      End  Sectors   Size Type
/dev/vda1  131072 29360122 29229051 111.5G Linux filesystem
/dev/vda2   16384   131071   114688   448M EFI System

Partition table entries are not in disk order.

root@morn /etc/grub.d # mount |grep /dev/vda
/dev/vda1 on / type ext4 (rw,relatime,quota,usrquota,grpquota,errors=remount-ro)
So far, we once again see without any doubt that the root file system ('/') is mounted on vda1 and that it is an ext4 file system with UUID 7c91164d-298d-4ef8-9823-df48a13e5325. Furthermore, the two file systems on vda definitely have different UUIDs. And now GRUB seems to have a massive bug:
root@morn /etc/grub.d # grub-probe -d /dev/vda1; grub-probe -d /dev/vda2
fat
fat

root@morn /etc/grub.d # grub-probe -t fs_uuid -d /dev/vda1; grub-probe -t fs_uuid -d /dev/vda2
4963-B5C0
4963-B5C0

root@morn /etc/grub.d # grub-probe /; grub-probe -t fs_uuid /
fat
4963-B5C0
So GRUB obviously has come to the conclusion that both file systems (on vda1 and vda2, respectively) are FAT, that they both have the same UUID (4963-B5C0), and that / is mounted on a FAT file system with that UUID. Of course, this is complete nonsense and clearly contradicts the output of blkid, mount and fdisk. Any ideas? **EDIT / UPDATE 2025-08-08 #1** I'd like to draw everybody's attention to something that my be easily missed when reading the post: As fdisk -l /dev/vda shows, the disk's physical sector size as well the disk's logical sector size are 4096 in that VM. As far as I can tell, this is a quite unusual scenario. We had our reasons to set it up that way, though, and we don't want to change it. Perhaps grub-probe is not able to deal with logical sector sizes other than 512. Can somebody confirm this?
Asked by Binarus (3901 rep)
Aug 6, 2025, 06:26 PM
Last activity: Aug 8, 2025, 06:53 AM