Failed to boot Debian server + slow in emergency mode
0
votes
0
answers
140
views
I am running Debian GNU/Linux 12.5.0 (bookworm) on a computer which has suddenly failed to boot and is extremely slow in emergency mode. Apologies in advance if I leave something important out, I am a Linux noob.
## Background ##
The problem began while running qBittorrent-nox, whereupon I realized after a while that the web-client was not responding. Investigating further, I also noticed that tries to connect to the server with ssh timed out. I held in the power button after this in an attempt to do a hard restart, which in retrospect should probably not have been done.
Powering up the machine again took a very long time (~5 minutes), which landed me in emergency mode with some error messages (see codebox in following chapter). While in emergency mode, every command took at least 10 seconds to respond, if not more. Soft restarting sucessfully booted the machine once, but the slowdown persisted. Every new soft restart after that has been into emergency mode.
I tried running in GRUB, but it got permanently stuck on some line and did not boot.
## Tried fixes ##
I've tried to run fsck -f on the EFI partition while in rescue mode. This seemingly did not do anything.
I've tried running a shell on /dev/nvme0n1p1 while in rescue mode but was notified that there was no shell available on this partition.
I've tried adding
Storage=persistent
to journald.conf as was suggested in but this did not do anything.
I've looked into changing the file system type, as in , but decided against it since it looks like I have the correct file system type.
I am currently out of ideas on what to look out for. Any help or input would be greatly appreciated.
Thank you in advance!
## Terminal responses ##
Upon starting the server I am greeted with the following text:
[ 0.074473] x86/cpu: SGX disabled by BIOS.
/dev/nvme0n1p2: clean, 88626/31162368 files, 75905231/124645632 blocks
[ TIME ] Timed out waiting for device dev-dis...E5.device - /dev/disk/by-uuid/1002
[DEPEND] Dependency failed for systemd-fsck@d...System Check on /dev/disk/by-uuid/
[DEPEND] Dependency failed for boot-efi.mount - /boot/efi.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.
[ TIME ] Timed out waiting for device dev-dis...y-uuid/f7809ea0-31f9-4370-ac28-531
[DEPEND] Dependency failed for dev-disk-by\x2...y-uuid/f7809ea0-31f9-4370-ac28-531
[DEPEND] Dependency failed for swap.target - Swaps.
[FAILED] Failed to start systemd-journal-flus...e - Flush Journal to Persistent St
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" ot boot into default mode.
Give root password for maintenance
(or press Control-D to continue):
I ran journalctl -xb, but the text file is large (1600 lines) and I do not know what's relevant in it. Please let me know if and how I should include it in the question.
Here's the response when running fdisk -l:
### fdisk -l ###
Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: INTEL SSDPEKNU512G8L
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F4FA719B-6D68-4615-9F20-031696FFE5CD
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1050623 1048576 512M EFI System
/dev/nvme0n1p2 1050624 998215679 997165056 475.5G Linux filesystem
/dev/nvme0n1p3 998215680 1000214527 1998848 976M Linux swap
Disk /dev/sda: 28.82 GiB, 30943995904 bytes, 60437492 sectors
Disk model: DataTraveler 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x03e7f784
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 60437471 60435424 28.8G c W95 FAT32 (LBA)
Something that I noticed here is that my sda or sdb partitions were seemingly gone. /dev/sda1 Here is a usb stick that I use both as a recovery device and a way to transfer log files from the linux device to my home desktop.
Here's the response when running systemctl status systemd-journald-service:
### status systemd-journald-service ###
● systemd-journald.service - Journal Service
Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
Active: active (running) since Thu 2024-05-30 13:07:03 CEST; 34min ago
TriggeredBy: ● systemd-journald-dev-log.socket
● systemd-journald.socket
● systemd-journald-audit.socket
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 718 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 9391)
Memory: 5.4M
CPU: 104ms
CGroup: /system.slice/systemd-journald.service
└─718 /lib/systemd/systemd-journald
May 30 13:07:03 SkibidiServer systemd-journald: File /var/log/journal/7063fd07468243ce889ee166584c6331/system.journal corrupted or uncleanly shut down, renaming and replacing.
May 30 13:07:03 SkibidiServer systemd-journald: Journal started
May 30 13:07:03 SkibidiServer systemd-journald: System Journal (/var/log/journal/7063fd07468243ce889ee166584c6331) is 184.0M, max 4.0G, 3.8G free.
May 30 13:06:30 SkibidiServer systemd: systemd-journald.service: Watchdog timeout (limit 3min)!
May 30 13:06:30 SkibidiServer systemd: systemd-journald.service: Killing process 276 (systemd-journal) with signal SIGABRT.
Here's the content of /etc/systemd/journald.conf:
### /etc/systemd/journald.conf ###
[Journal]
Storage=persistent
Here's the content of /etc/fstab:
### /etc/fstab ###
# / was on /dev/nvme0n1p2 during installation
UUID=6f0758b8-fd39-45c4-a0ed-e3cb67144a1c / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=1002-2BE5 /boot/efi vfat umask=0077 0 1
# swap was on /dev/nvme0n1p3 during installation
UUID=f7809ea0-31f9-4370-ac28-5310ffae8074 none swap sw 0 0
And finally, here's the response when running blkid
/dev/nvme0n1p3: UUID="f7809ea0-31f9-4370-ac28-5310ffae8074" TYPE="swap" PARTUUID="797058e6-c6cf-476e-aad9-7a99bddd79d5"
/dev/nvme0n1p1: UUID="1002-2BE5" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="519bfa3c-68d2-412d-bdea-b97f51a49d02"
/dev/nvme0n1p2: UUID="6f0758b8-fd39-45c4-a0ed-e3cb67144a1c" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="ea266a1a-45da-4e66-a399-6625f2b24259"
/dev/sda1: LABEL_FATBOOT="DEBIAN12_5_" LABEL="DEBIAN12_5_" UUID="22DD-A0FE" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="03e7f784-01"
Edit: Upon request, I've run fsck on all devices. Here's the response:
### fsck -f /dev/nvme0n1p1 ###
fsck from util-linux 2.38.1
fsck.fat 4.2 (2021-01-31)
/dev/nvme0n1p1: 8 files, 1493/130812 clusters
### fsck -f -y /dev/nvme0n1p2 ###
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Inode 6946841 extent tree (at level 1) could be narrower. Optimize? yes
...A bunch of the same prompts which I answered yes to all, inode number was always 69468xx, varying between level 1 and 2...
.
Inode 12976130 extent tree (at level 2) could be narrower. Optimize? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/nvme0n1p2: 88632/31162368 files (0.3% non-contiguous), 75909135/124645632 blocks
After running this, the statement [FAILED] Failed to start systemd-journal-flus...e - Flush Journal to Persistent St
has dissapeared when booting, but it still boots to emergency mode and is extremely slow.
### fsck -f /dev/nvme0n1p3 ###
fsck from util-linux 2.38.1
Asked by Isak Lundgren
(1 rep)
May 30, 2024, 12:25 PM
Last activity: May 31, 2024, 09:49 AM
Last activity: May 31, 2024, 09:49 AM