Sample Header Ad - 728x90

Failed to boot Debian server + slow in emergency mode

0 votes
0 answers
140 views
I am running Debian GNU/Linux 12.5.0 (bookworm) on a computer which has suddenly failed to boot and is extremely slow in emergency mode. Apologies in advance if I leave something important out, I am a Linux noob. ## Background ## The problem began while running qBittorrent-nox, whereupon I realized after a while that the web-client was not responding. Investigating further, I also noticed that tries to connect to the server with ssh timed out. I held in the power button after this in an attempt to do a hard restart, which in retrospect should probably not have been done. Powering up the machine again took a very long time (~5 minutes), which landed me in emergency mode with some error messages (see codebox in following chapter). While in emergency mode, every command took at least 10 seconds to respond, if not more. Soft restarting sucessfully booted the machine once, but the slowdown persisted. Every new soft restart after that has been into emergency mode. I tried running in GRUB, but it got permanently stuck on some line and did not boot. ## Tried fixes ## I've tried to run fsck -f on the EFI partition while in rescue mode. This seemingly did not do anything. I've tried running a shell on /dev/nvme0n1p1 while in rescue mode but was notified that there was no shell available on this partition. I've tried adding Storage=persistent to journald.conf as was suggested in but this did not do anything. I've looked into changing the file system type, as in , but decided against it since it looks like I have the correct file system type. I am currently out of ideas on what to look out for. Any help or input would be greatly appreciated. Thank you in advance! ## Terminal responses ## Upon starting the server I am greeted with the following text:
[    0.074473] x86/cpu: SGX disabled by BIOS.
/dev/nvme0n1p2: clean, 88626/31162368 files, 75905231/124645632 blocks
[ TIME ] Timed out waiting for device dev-dis...E5.device - /dev/disk/by-uuid/1002
[DEPEND] Dependency failed for systemd-fsck@d...System Check on /dev/disk/by-uuid/
[DEPEND] Dependency failed for boot-efi.mount - /boot/efi.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.
[ TIME ] Timed out waiting for device dev-dis...y-uuid/f7809ea0-31f9-4370-ac28-531
[DEPEND] Dependency failed for dev-disk-by\x2...y-uuid/f7809ea0-31f9-4370-ac28-531
[DEPEND] Dependency failed for swap.target - Swaps.
[FAILED] Failed to start systemd-journal-flus...e - Flush Journal to Persistent St
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" ot boot into default mode.
Give root password for maintenance
(or press Control-D to continue):
I ran journalctl -xb, but the text file is large (1600 lines) and I do not know what's relevant in it. Please let me know if and how I should include it in the question. Here's the response when running fdisk -l: ### fdisk -l ###
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: INTEL SSDPEKNU512G8L                    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F4FA719B-6D68-4615-9F20-031696FFE5CD

Device             Start        End   Sectors   Size Type
/dev/nvme0n1p1      2048    1050623   1048576   512M EFI System
/dev/nvme0n1p2   1050624  998215679 997165056 475.5G Linux filesystem
/dev/nvme0n1p3 998215680 1000214527   1998848   976M Linux swap


Disk /dev/sda: 28.82 GiB, 30943995904 bytes, 60437492 sectors
Disk model: DataTraveler 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x03e7f784

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1  *     2048 60437471 60435424 28.8G  c W95 FAT32 (LBA)
Something that I noticed here is that my sda or sdb partitions were seemingly gone. /dev/sda1 Here is a usb stick that I use both as a recovery device and a way to transfer log files from the linux device to my home desktop. Here's the response when running systemctl status systemd-journald-service: ### status systemd-journald-service ###
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Thu 2024-05-30 13:07:03 CEST; 34min ago
TriggeredBy: ● systemd-journald-dev-log.socket
             ● systemd-journald.socket
             ● systemd-journald-audit.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 718 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 9391)
     Memory: 5.4M
        CPU: 104ms
     CGroup: /system.slice/systemd-journald.service
             └─718 /lib/systemd/systemd-journald

May 30 13:07:03 SkibidiServer systemd-journald: File /var/log/journal/7063fd07468243ce889ee166584c6331/system.journal corrupted or uncleanly shut down, renaming and replacing.
May 30 13:07:03 SkibidiServer systemd-journald: Journal started
May 30 13:07:03 SkibidiServer systemd-journald: System Journal (/var/log/journal/7063fd07468243ce889ee166584c6331) is 184.0M, max 4.0G, 3.8G free.
May 30 13:06:30 SkibidiServer systemd: systemd-journald.service: Watchdog timeout (limit 3min)!
May 30 13:06:30 SkibidiServer systemd: systemd-journald.service: Killing process 276 (systemd-journal) with signal SIGABRT.
Here's the content of /etc/systemd/journald.conf: ### /etc/systemd/journald.conf ###
[Journal]
Storage=persistent
Here's the content of /etc/fstab: ### /etc/fstab ###
# / was on /dev/nvme0n1p2 during installation
UUID=6f0758b8-fd39-45c4-a0ed-e3cb67144a1c /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=1002-2BE5  /boot/efi       vfat    umask=0077      0       1
# swap was on /dev/nvme0n1p3 during installation
UUID=f7809ea0-31f9-4370-ac28-5310ffae8074 none            swap    sw              0       0
And finally, here's the response when running blkid
/dev/nvme0n1p3: UUID="f7809ea0-31f9-4370-ac28-5310ffae8074" TYPE="swap" PARTUUID="797058e6-c6cf-476e-aad9-7a99bddd79d5"
/dev/nvme0n1p1: UUID="1002-2BE5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="519bfa3c-68d2-412d-bdea-b97f51a49d02"
/dev/nvme0n1p2: UUID="6f0758b8-fd39-45c4-a0ed-e3cb67144a1c" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea266a1a-45da-4e66-a399-6625f2b24259"
/dev/sda1: LABEL_FATBOOT="DEBIAN12_5_" LABEL="DEBIAN12_5_" UUID="22DD-A0FE" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="03e7f784-01"
Edit: Upon request, I've run fsck on all devices. Here's the response: ### fsck -f /dev/nvme0n1p1 ###
fsck from util-linux 2.38.1
fsck.fat 4.2 (2021-01-31)
/dev/nvme0n1p1: 8 files, 1493/130812 clusters
### fsck -f -y /dev/nvme0n1p2 ###
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Inode 6946841 extent tree (at level 1) could be narrower. Optimize? yes
...A bunch of the same prompts which I answered yes to all, inode number was always 69468xx, varying between level 1 and 2...
.
Inode 12976130 extent tree (at level 2) could be narrower. Optimize? yes

Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p2: 88632/31162368 files (0.3% non-contiguous), 75909135/124645632 blocks
After running this, the statement [FAILED] Failed to start systemd-journal-flus...e - Flush Journal to Persistent St has dissapeared when booting, but it still boots to emergency mode and is extremely slow. ### fsck -f /dev/nvme0n1p3 ###
fsck from util-linux 2.38.1
Asked by Isak Lundgren (1 rep)
May 30, 2024, 12:25 PM
Last activity: May 31, 2024, 09:49 AM