I have Btrfs setup on 3 disks with metadata and data in RAID1. But now I have a checksum error which it cannot recover. The checksum is the same on both copies and only differs from the expected checksum by one flipped bit. Therefore I suspect there was a bitflip on the checksum before it was written to the disks (the computer does not have ECC RAM). I have a copy of the actual file on another computer from before it was written to this filesystem but as shown below I cannot read out the data due to I/O error from the filesystem so I cannot compare them. How should I go on to fix this error? Some details: $ uname -a Linux stan 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ btrfs --version btrfs-progs v4.15.1 $ sudo btrfs fi usage /media/btrfs/ Overall: Device size: 7.28TiB Device allocated: 3.91TiB Device unallocated: 3.36TiB Device missing: 0.00B Used: 3.83TiB Free (estimated): 1.72TiB (min: 1.72TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,RAID1: Size:1.95TiB, Used:1.91TiB /dev/sdb 1.95TiB /dev/sdc 998.00GiB /dev/sdd 1001.00GiB Metadata,RAID1: Size:4.00GiB, Used:2.63GiB /dev/sdb 4.00GiB /dev/sdc 3.00GiB /dev/sdd 1.00GiB System,RAID1: Size:64.00MiB, Used:304.00KiB /dev/sdb 64.00MiB /dev/sdc 64.00MiB Unallocated: /dev/sdb 1.68TiB /dev/sdc 861.95GiB /dev/sdd 861.02GiB Scrub: $ sudo btrfs scrub status /media/btrfs/ scrub status for xxxxxx scrub started at Mon Aug 24 11:23:27 2020 and finished after 03:41:54 total bytes scrubbed: 3.81TiB with 2 errors error details: csum=2 corrected errors: 0, uncorrectable errors: 2, unverified errors: 0 Dmesg error after scrub. $ dmesg ... 196755.786038] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5223, inod e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2) [196755.786168] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5303, inod e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2) [196755.786245] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5302, inod e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2) ... [196755.788274] BTRFS error (device sdb): bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 [196755.814044] BTRFS error (device sdb): unable to fixup (regular) error at logical 3099310968832 on dev /dev/sdb Inspect-internal on block: $ sudo btrfs inspect-internal logical-resolve -v 3099310968832 /media/btrfs/ ioctl ret=0, total_size=4096, bytes_left=3456, bytes_missing=0, cnt=78, missed=0 ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0 /media/btrfs//snapshots/stansafe.20200601T032501+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2 ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0 /media/btrfs//snapshots/stansafe.20200910T032501+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2 ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0 /media/btrfs//snapshots/stansafe.20200909T032502+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2 ... Trying to verify the file: $ sha256sum /media/btrfs//stansafe/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2 sha256sum: /media/btrfs//stansafe/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2: Input/output error $ dmesg ... [1642985.509498] BTRFS warning (device sdb): csum failed root 259 ino 6521311 off 7614464 csum 0x151ad4ce expected csum 0x150ad4ce mirror 1 [1642985.509942] BTRFS warning (device sdb): csum failed root 259 ino 6521311 off 7614464 csum 0x151ad4ce expected csum 0x150ad4ce mirror 2

Similar to here , but no solution was found for them either. I also tried checking the history, see if something was running 'rm' but no dice. Periodically, my device stops responding. The *only* commands I can use are builtin shell commands such as cd, echo, pwd. Any other command such as ls (bin) or fsck (sbin)... all show -bash: /usr/bin/ls: no such file or directory, yet the files still exist. If I cd into /usr/bin and run echo *, I see a list of all the commands that supposedly cannot be found. ls, sudo, ect... , but I cannot execute them. Attempting to manually execute /usr/bin/ls results in the same error. I checked my path variable... it seemed fine echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin The only files I noticed missing were usr/bin/init and usr/sbin/init (thus prompting the boot to initramfs on restart). Additionally, under root / the /bin and /sbin folders were *gone*, possibly the symlinks were missing? On a working install, there is a symlink between /bin -> /usr/bin When this occurs, since I have no permissions, and nothing else works, I am forced to restart by powering off/on the machine, followed by it booting to initramfs. I know a temporary fix to the the issue... From an USB with a fresh install of my distribution, I copy over /bin, /sbin, /lib to my boot partition under /root. But this doesn't solve it permanently, and the issue comes back after a while. I would really appreciate any help, I'm honestly completely lost as to what to do now. I have tried... *1. fsck the partitions.* Both my boot partition (mmcblk0p1) & root partition (mmcblk0p2) seem fine. Dirty bit was set on boot partition 0p1, likely due to hard poweroff. 2nd pass determined no issues. *2. badblocks* found 0 bad blocks in its read test *3. Memtest the device* I haven't been able to memtest it yet, as I cannot figure out how to get my device to boot to it. I flashed memtest to an SD card, as well as a usb. My device won't boot from either. I don't have a bios, it's running u-boot. Supposedly, I can interrupt the u-boot process to select where to boot, but pressing keys does nothing during boot. Someone here suggests it may be that there isn't USB support for the keyboard in u-boot, only once it boots to linux... making it impossible to boot from another drive? The device is an Orange Pi 5B, with Ubuntu 22.04 flashed to the EMMC drive built onto the device. *4. Test the PSU* I am running with this 5v 4a PSU, people Here seem to have similar issues. How can I test if my PSU is the issue, or if nothing else, can someone suggest one that wont have problems, so I can rule the PSU issue out?

# Goal I am trying to figure out why my file system has become read-only so I can address any potential hardware or security issues (main concern) and maybe fix the issue without having to reinstall everything and migrate my files from backup (I might lose some data but probably not much). According to the manual of btrfs check: > Do not use --repair unless you are advised to do so by a developer or an experienced user, and then only after having accepted that no fsck successfully repair all types of filesystem corruption. Eg. some other software or hardware bugs can fatally damage a volume. I am thinking of trying the --repair option or btrfs scrub but want input from a more experienced user. # What I’ve tried I first noticed a read-only file system when trying to update my system in the terminal. I was told: Cannot open log file: (30) - Read-only file system [/var/log/dnf5. log] I have run basic checks (using at least 3 different programs) of my SSD without anything obviously wrong. The SSD and everything else in my computer is about 6 and a half years old, so maybe something is failing. Here is the SMART Data section of the output from sudo smartctl -a /dev/nvme0n1:

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0x1)
Critical Warning: 0x00
Temperature: 31 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 1%
Data Units Read: 33,860,547 [17.3 TB]
Data Units Written: 31,419,841 [16.0 TB]
Host Read Commands: 365,150,063
Host Write Commands: 460,825,882
Controller Busy Time: 1,664
Power Cycles: 8,158
Power On Hours: 1,896
Unsafe Shutdowns: 407
Media and Data Integrity Errors: 0
Error Information Log Entries: 4,286
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 31 Celsius
Temperature Sensor 2: 30 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06, NSID Oxffffffff)
Self-test status: No self-test in progress
No Self-tests Logged

I tried the following I think from a live disk sudo mount -o remount,rw /mount/point but that output an error such as, cannot complete read-only system. sudo btrfs device stats /home **and** sudo btrfs device stats / outputs:

[/dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d].write_io_errs 0
[/dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d].read_io_errs 0
[/dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d].flush_io_errs 0
[/dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d].corruption_errs 14
[/dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d].generation_errs 0

**This seems to suggest that corruption is only in the /home directory.** **However, sudo btrfs check /dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d stops at [5/8] checking fs roots with the end of the output at the top of this image:** sudo btrfs check /dev/mapper/luks-7215db73-54d1-437e-875d-f82fae508b5d" class="img-fluid rounded" style="max-width: 100%; height: auto; margin: 10px 0;" loading="lazy"> **Some of these files may be in the / directory, but I’m not sure without looking into further.** sudo btrfs fi usage / provides: sudo btrfs fi usage /" class="img-fluid rounded" style="max-width: 100%; height: auto; margin: 10px 0;" loading="lazy"> **I think that Data,single, Metadata,DUP, and System,DUP might be saying I can repair the corruption if it’s only in metadata or system but not if it’s the actual file data. Might be something to explore more.** Here is vi /etc/fstab: vi /etc/fstab" class="img-fluid rounded" style="max-width: 100%; height: auto; margin: 10px 0;" loading="lazy"> sudo dmesg | grep -i “btrfs” states: sudo dmesg | grep -i “btrfs”" class="img-fluid rounded" style="max-width: 100%; height: auto; margin: 10px 0;" loading="lazy"> The file system is indeed unstable. Once, I wasn’t able to list any files in my /home directory, but I haven't run into this issue again across several reboots. # What I think might be causing this I suspect that changing my username, hostname, and display name (shown on the login screen) recently may have caused problems because my file system became read-only about a week to a week and a half after doing so. I followed some tutorials online, but I noticed that many of my files still had the group and possibly user belonging to the old username. So I created a symbolic link at the top of my home directory pointing the old username to the new one, and it seemed like everything was fine until the read-only issue. There may have been more I did, but I don’t remember exactly as it’s been a few weeks now. I have a history of most or all of the commands I ran if it might be helpful. I think it may be something hardware related, something I did, software bugs (maybe introduced by a recent update — I have a picture of packages affected in my most recent dnf upgrade transaction, but I was unable to rollback or undo the upgrade because of the read-only file system), improper shutdowns (may have done this while making changes to the username, hostname, and display name), or a security issue.

For some unknown reason, there are 2 directories I can't delete. First directory called **sw.old** is empty and can be deleted only by rm, as rmdir won't recognize it. However, even after rm, it still shows up: [02:11:36] user@user:/media/user/exthdd/docs$ ls -il total 1072064 1456 drwx------ 1 user user 0 Aug 12 10:04 1old.or.probably.unfinished 5717 drwx------ 1 user user 8192 Jan 27 22:58 videos 6528 -rw------- 1 user user 1097779088 Nov 5 16:15 release_Remix_OS_for_PC_Android_M_64bit_B2016112101.zip 8008 drwx------ 1 user user 4096 Jan 28 00:55 txt 64 drwx------ 1 user user 0 Dec 25 22:15 sw.old [02:12:03] user@user:/media/user/exthdd/docs$ rmdir sw.old/ rmdir: failed to remove ‘sw.old/’: No such file or directory [02:12:57] user@user:/media/user/exthdd/docs$ rm -rf sw.old/ [02:13:15] user@user:/media/user/exthdd/docs$ ls -il total 1072064 1456 drwx------ 1 user user 0 Aug 12 10:04 1old.or.probably.unfinished 5717 drwx------ 1 user user 8192 Jan 27 22:58 videos 6528 -rw------- 1 user user 1097779088 Nov 5 16:15 release_Remix_OS_for_PC_Android_M_64bit_B2016112101.zip 8008 drwx------ 1 user user 4096 Jan 28 00:55 txt 64 drwx------ 1 user user 0 Dec 25 22:15 sw.old Second one called **misc** has a corrupted file inside it: [02:24:32] user@user:/media/user/exthdd/docs/txt$ ls -il total 0 22607 drwx------ 1 user user 0 Dec 31 16:09 misc [02:24:36] user@user:/media/user/exthdd/docs/txt$ ls -il misc/ ls: cannot access misc/patterns.mp4: Input/output error total 0 ? -????????? ? ? ? ? ? patterns.mp4 [02:24:54] user@user:/media/user/exthdd/docs/txt$ rm -rf misc/ rm: cannot remove ‘misc/patterns.mp4’: Input/output error How can I remove those directories (and corrupted file inside one of them) without formatting?

I have a VM which has an attached raided device with fstab entry: /dev/md127 /mnt/blah ext4 nofail 0 2 The raided disks are corrupted and during startup the unit entered emergency/maintence mode, which means only the local host user could exit this mode and start it up normally. During normal startup the following occurred in syslog: systemd-fsck: /dev/md127 contains a file system with errors, check forced. systemd-fsck: /dev/md127: Inodes that were part of a corrupted orphan linked list found. systemd-fsck: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. systemd-fsck: #011(i.e., without -a or -p options) systemd-fsck: fsck failed with exit status 4. systemd-fsck: Running request emergency.target/start/replace systemd: systemd-fsck@dev-md127.service: Main process exited, code=exited, status=1/FAILURE systemd: systemd-fsck@dev-md127.service: Failed with result 'exit-code'. systemd: Failed to start File System Check on /dev/md127. systemd: Dependency failed for /mnt/blah. systemd: Dependency failed for Provisioner client daemon. My guess is that the OS goes to emergency/maintenance mode because of the corrupt raided disks: systemctl --state=failed UNIT LOAD ACTIVE SUB DESCRIPTION ● systemd-fsck@dev-md127.service loaded failed failed File System Check on /dev/md127 What i want is for the VM to startup regardless of whether the raided drives are corrupt/unmountable, so it shouldn't go to emergency/maintenance mode. I followed these posts to attempt at disabling emergency/maintenance mode: - https://unix.stackexchange.com/questions/416640/how-to-disable-systemd-agressive-emergency-shell-behaviour - https://unix.stackexchange.com/questions/326493/how-to-determine-exactly-why-systemd-enters-emergency-mode/393711#393711 - https://unix.stackexchange.com/questions/422319/emergency-mode-and-local-disk I had to first create the directory local-fs.target.d in /etc/systemd/system/, which felt wrong. I then created a nofail.conf in /etc/systemd/system/local-fs.target.d/nofail.conf containing: [Unit] OnFailure= After loading that drop file, I was able to confirm that the drop file was found by local-fs.target: sudo systemctl status local-fs.target ● local-fs.target - Local File Systems Loaded: loaded (/lib/systemd/system/local-fs.target; static; vendor preset: enabled) Drop-In: /etc/systemd/system/local-fs.target.d └─nofail.conf Active: active since Tue 2019-01-08 12:36:41 UTC; 3h 55min ago Docs: man:systemd.special(7) BUT, after rebooting, the VM still ended up in emergency/maintenance mode. Have i missed something? Does the nofail.conf solution not work with raided disks? ---------- EDIT: I was able to get a print out of the logs when the system booted to emergency mode (sorry it's a screenshot since i don't have access to the host and had to ask the owner for it): Here's the output from systemctl for systemd-fsck@dev-md127:

sudo systemctl status --no-pager --full systemd-fsck@dev-md127
 ● systemd-fsck@dev-md127.service - File System Check on /dev/md127
    Loaded: loaded (/lib/systemd/system/systemd-fsck@.service; static; vendor preset: enabled)
    Active: failed (Result: exit-code) since Thu 2019-01-10 12:05:44 UTC; 2h 57min ago
      Docs: man:systemd-fsck@.service(8)
   Process: 1025 ExecStart=/lib/systemd/systemd-fsck /dev/md127 (code=exited, status=1/FAILURE)
  Main PID: 1025 (code=exited, status=1/FAILURE)

 systemd: Starting File System Check on /dev/md127...
 systemd-fsck: /dev/md127 contains a file system with errors, check forced.
 systemd-fsck: /dev/md127: Inodes that were part of a corrupted orphan linked list found.
 systemd-fsck: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
 systemd-fsck:         (i.e., without -a or -p options)
 systemd-fsck: fsck failed with exit status 4.
 systemd-fsck: Running request emergency.target/start/replace
 systemd: systemd-fsck@dev-md127.service: Main process exited, code=exited, status=1/FAILURE
 systemd: systemd-fsck@dev-md127.service: Failed with result 'exit-code'.
 systemd: Failed to start File System Check on /dev/md127.

As i pointed out earlier, i have nofail set in /etc/fstab. Now the questions are: 1. What is the dependency in the failed dependency in the screenshot? 2. If fsck fails on /dev/md127, why does it enter emergency mode and how do i disable that? EDIT 2: A couple of other things i can add are: 1. the vm is a kvm vm 2. it's a software raid Kind regards, Ankur

I'm new to Linux (just over 1 month) and I’ve encountered a problem with my drive. I have the main drive (SSD) and a secondary drive (HDD). The HDD was working fine, but now I’m getting this error message:

An error occurred while accessing 'New Volume', the system responded:
The requested operation has failed: 
Error mounting /dev/sda1 at /run/media/spicypenguin/New Volume: 
wrong fs type, bad option, bad superblock on /dev/sda1, missing codepage or helper program, or other error

And now my HDD is not accessible. I’ve tried rebooting, upgrading the system, etc., but none of these solutions worked. I’m using CachyOS:latest (Arch-based).

Woke up to a nasty surprise a couple of days ago to see that my zpool couldn't be imported. After discovering that sdb had faulted I assumed it was time to replace the drive, so I hot swapped it. After trying to figure out what to do next for a while zfs commands became non responsive, I assumed that maybe it was trying to repair itself, left it running for over a day, since zfs commands were still unresponsive I then force restarted the server. Now zpool import shows sdb twice... pool FAULTED corrupted data raidz1-0 DEGRADED sda ONLINE sdb FAULTED corrupted data sdb ONLINE sde ONLINE Can anyone tell me what is going on, and what the best way is to repair the pool? I have not yet checked the drive that faulted.

I tried to update a server over SSH, but when I ran yum check-update I got an error: > [error 5] Input/output error I think this means the RPM libraries may be damaged or corrupt, but I'm not sure how to resolve this.

**TL;DR summary**: Translate an md sector number into offsets(s) within the /dev/mdX device, and how to investigate it with xfs_db. The sector number is from sh->sector in linux/drivers/md/raid5.c:handle_parity_checks5() . I don't know MD internals, so I don't know exactly what to do with the output from the printk logging I added. Offsets into the component devices (for dd or a hex editor/viewer) would also be interesting. I suppose I should ask this on the Linux-raid mailing list. Is it subscribers-only, or can I post without subscribing? ---- I have xfs directly on top of MD RAID5 of 4 disks in my desktop (no LVM). A recent scrub detected a non-zero mismatch_cnt (8 in fact, because md operates on 4kiB pages at a time). This is a RAID5, not RAID1/RAID10 where mismatch_cnt != 0 can happen during normal operation . (The other links at the bottom of this wiki page might be useful to some people.) I could just blindly repair, but then I'd have no idea which file to check for possible corruption, besides losing any chance to choose which way to reconstruct. Frostschutz's answer on a similar question is the only suggestion I found for tracking back to a difference in the filesystem. It's cumbersome and slow, and I'd rather use something better to narrow it down to a few files first. --- ### Kernel patch to add logging Bizarrely, md's check feature doesn't report where an error was found . **I added a printk in md/raid5.c to log sh->sector in the if branch that increments mddev->resync_mismatches in handle_parity_checks5()** (tiny patch published on github , originally based on 4.5-rc4 from kernel.org.) For this to be ok for general use, it would probably need to avoid flooding the logs in repairs with a lot of mismatches (maybe only log if the new value of resync_mismatches is sector * 512 a linear address inside /dev/md/t-r5 (aka /dev/md125)? Is it a sector number within each component device (so it refers to three data and one parity sector)? I'm guessing the latter, since a parity-mismatch in RAID5 means N-1 sectors of the md device are in peril, offset from each other by the stripe unit. Is sector 0 the very start of the component device, or is it the sector after the superblock or something? Was there more information in handle_parity_checks5()` that I should have calculated / logged? If I wanted to get just the mismatching blocks, is this correct? dd if=/dev/sda6 of=mmblock.0 bs=512 count=8 skip=4294708224 dd if=/dev/sdb6 of=mmblock.1 bs=512 count=8 skip=4294708224 dd if=/dev/sda6 of=mmblock.2 bs=512 count=8 skip=4294708224 dd if=/dev/sdd of=mmblock.3 bs=512 count=8 skip=4294708224 ## not a typo: my 4th component is a smaller full-disk # i.e. sec_block() { for dev in {a,b,c}6 d; do dd if=/dev/sd"$dev" of="sec$1.$dev" skip="$1" bs=512 count=8;done; }; sec_block 123456 I'm guessing not, because I get 4k of zeros from all four raid components, and 0^0 == 0, so that should be the correct parity, right? One other place I've seen mention of using sector addresses in md is for sync_min and sync_max (in sysfs). Neil Brown on the linux-raid list , in response to a question about a failed drive with sector numbers from hdrecover, where Neil used the full-disk sector number as an MD sector number. That's not right is it? Wouldn't md sector numbers be relative to the component devices (partitions in that case), not the full device that the partition is a part of? ---- ### linear sector to XFS filename: Before realizing that the md sector number was probably for the components, not the RAID device, I tried using it in read-only xfs_db: Dave Chinner's very brief suggestion on how to find how XFS is using a given block didn't seem to work at all for me. (I would have expected some kind of result, for some sector, since the number shouldn't be beyond the end of the device even if it's not the mismatched sector) # xfs_db -r /dev/md/t-r5 xfs_db> convert daddr 4294708224 fsblock 0x29ad5e00 (699227648) xfs_db> blockget -nv -b 699227648 xfs_db> blockuse -n # with or without -c 8 must run blockget first huh? What am I doing wrong here? I guess this should be a separate question. I'll replace this with a link if/when I ask it or find an answer to this part somewhere else. My RAID5 is essentially idle, with no write activity and minimal read (and noatime, so reads aren't producing writes). ---- ### Extra stuff about my setup, nothing important here Many of my files are video or other compressed data that give an effective way to tell whether the data is correct or not (either internal checksums in the file format, or just whether it decodes without errors). That would make this read-only loopback method viable, once I know which file to check. I didn't want to run a 4-way diff of every file in the filesystem to find the mismatch first, though, when the kernel has the necessary information while checking, and could easily log it. ---- my /proc/mdstat for my bulk-data array: md125 : active raid5 sdd[3] sda6 sdb6[1] sdc6[4] 7325273088 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/19 pages [0KB], 65536KB chunk It's on partitions on three Toshiba 3TB drives, and a non-partitioned WD25EZRS green-power (slow) drive which I'm replacing with another Toshiba. (Using [mdadm --replace](https://unix.stackexchange.com/a/104052/79808) to do it online with no gaps in redundancy. I realized after one copy that I should check the RAID health before as well as after, to detect problems. That's when I detected the mismatch. It's possible it's been around for a long time, since I had some crashes almost a year ago, but I don't have old logs and mdadm doesn't seem to send mail about this by default (Ubuntu 15.10). My other filesystems are on RAID10f2 devices made from earlier partitions on the three larger HDs (and RAID0 for /var/tmp). The RAID5 is just for bulk-storage, not /home or /. My drives are all fine: SMART error counts are 0 all bad-block counters on all drives, and short + long SMART self-tests passed. ---- near-duplicates of this question which don't have answers: * https://unix.stackexchange.com/questions/256514/what-chunks-are-mismatched-in-a-linux-md-array * http://www.spinics.net/lists/raid/msg49459.html * MDADM mismatch_cnt > 0. Any way to identify which blocks are in disagreement? * Other things already linked inline, but most notably frostschutz's read-only loopback idea . * scrubbing on the Arch wiki RAID page

This may sound stupid, and it is, I installed a distro directly into my flash drive, after checking everything worked I loaded kali liveUSB to try and format it back to it's original size, however none of my attempts seems to be working, below is a list of what I've tried so far: - Using gParted As of now trying to format using gParted I get the following error messages And This shows at the ending of the operation After that, running fdisk -l shows no external flash drives, then after removing and re-inserting it shows. - Using Data Description (dd) I already tried using Data Description but get the following error message

Lately when I write to an external drive formatted as NTFS* and then eject it, and wait a full minute before un-powering, then upon re-powering it won't auto-mount and manual mount attempts give an error. It wasn't like this before as it started around 2 months ago. An Ext4-formatted drive doesn't have this problem, and can even be hot-unplugged without issues (discovered by having a bad dock). Log excerpt: (Important point: $MFTMirr does not match $MFT)

Mar 10 14:42:39 confucius kernel: usb 4-1.1: new SuperSpeed USB device number 10 using xhci_hcd
Mar 10 14:42:39 confucius kernel: usb 4-1.1: New USB device found, idVendor=1058, idProduct=107c, bcdDevice=10.42
Mar 10 14:42:39 confucius kernel: usb 4-1.1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
Mar 10 14:42:39 confucius kernel: usb 4-1.1: Product: Elements 107C
Mar 10 14:42:39 confucius kernel: usb 4-1.1: Manufacturer: Western Digital
Mar 10 14:42:39 confucius kernel: usb 4-1.1: SerialNumber: 574343344E30353730323533
Mar 10 14:42:39 confucius kernel: usb-storage 4-1.1:1.0: USB Mass Storage device detected
Mar 10 14:42:39 confucius kernel: scsi host5: usb-storage 4-1.1:1.0
Mar 10 14:42:39 confucius mtp-probe: checking bus 4, device 10: "/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/usb4/4-1/4-1.1"
Mar 10 14:42:39 confucius mtp-probe: bus: 4, device: 10 was not an MTP device
Mar 10 14:42:39 confucius mtp-probe: checking bus 4, device 10: "/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/usb4/4-1/4-1.1"
Mar 10 14:42:39 confucius mtp-probe: bus: 4, device: 10 was not an MTP device
Mar 10 14:42:40 confucius kernel: scsi 5:0:0:0: Direct-Access     WD       Elements 107C    1042 PQ: 0 ANSI: 6
Mar 10 14:42:40 confucius kernel: sd 5:0:0:0: Attached scsi generic sg4 type 0
Mar 10 14:42:40 confucius kernel: sd 5:0:0:0: [sdd] Spinning up disk...
Mar 10 14:42:49 confucius kernel: .........ready
Mar 10 14:42:49 confucius kernel: sd 5:0:0:0: [sdd] 732558336 4096-byte logical blocks: (3.00 TB/2.73 TiB)
Mar 10 14:42:49 confucius kernel: sd 5:0:0:0: [sdd] Write Protect is off
Mar 10 14:42:49 confucius kernel: sd 5:0:0:0: [sdd] No Caching mode page found
Mar 10 14:42:49 confucius kernel: sd 5:0:0:0: [sdd] Assuming drive cache: write through
Mar 10 14:42:49 confucius kernel: sdd: sdd1
Mar 10 14:42:49 confucius kernel: sd 5:0:0:0: [sdd] Attached SCSI disk
Mar 10 14:42:50 confucius kernel: ntfs3: Max link count 4000
Mar 10 14:42:50 confucius kernel: ntfs3: Enabled Linux POSIX ACLs support
Mar 10 14:42:50 confucius kernel: ntfs3: Read-only LZX/Xpress compression included
Mar 10 14:43:05 confucius kernel: ntfs3: sdd1: volume is dirty and "force" flag is not set!
Mar 10 14:43:05 confucius udisksd: $MFTMirr does not match $MFT (record 3).
Mar 10 14:43:05 confucius udisksd: Failed to mount '/dev/sdd1': Input/output error
Mar 10 14:43:05 confucius udisksd: NTFS is either inconsistent, or there is a hardware fault, or it's a
Mar 10 14:43:05 confucius udisksd: SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
Mar 10 14:43:05 confucius udisksd: then reboot into Windows twice. The usage of the /f parameter is very
Mar 10 14:43:05 confucius udisksd: important! If the device is a SoftRAID/FakeRAID then first activate
Mar 10 14:43:05 confucius udisksd: it and mount a different device under the /dev/mapper/ directory, (e.g.
Mar 10 14:43:05 confucius udisksd: /dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
Mar 10 14:43:05 confucius udisksd: for more details.

Doing a disk check in Windows takes very long, I think in the order of 5 hours. This isn't a workable situation for us. Is there a way that I can troubleshoot this error in a reasonable manner (not taking 5 hours each time), or a way to just fix this by discarding/ignoring one of the $MFT? * drive shares data with Windows PCs --- Nemo has auto-mounting enabled. I eject with the eject button in Nemo (i.e. not umount)

My SSD died and all its data seems to be inaccessible and my Windows computer won't boot anymore. After going to a local computer store I was told that they could fix it by wiping all the data. I can wipe the drive myself and save 200$. But the only way to recover the information inside would be sending the SSD to a laboratory which they estimated would set be back around 500$. What truly matters to me is the data but I can't afford that lab price now. Would it be possible for me to backup the corrupted data to an external drive to be able to send that external drive to a lab once I can afford it? That way I could at least use again my computer in the meantime. Right now I can boot up the computer by using a ubuntu bootable usb as a temporary measure. After doing some research I tried

fdisk
sudo dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync status=progress

To copy the corrupted data to the external drive, but it did not work. Is there anything I can do or I should start accepting that all the data has been lost? As an extra info: I cannot install anything apt get on the computer right now due to WiFi stuff, I could do it I think if I made a more permanent installation of linux and started messing around with the connectivity settings. Also I remember that about half the SSD was empty, could I force a partition in that half to install linux or keep that half usable? Thanks!

While creating an install USB disk, I made the trivial error of indicating the wrong device, and ended up overwriting the initial few hundred megabytes of a 230 GB disk. The data was not extremely important, but I would still like to recover what I can. The first, obvious attempt, was `photorec`, which did find some stuff. But then, I hoped I could "fix" what was left of the filesystem (after all, I overwrote less than 0.5%). So I tried to run `e2fsck` with several combinations of parameters, including using a backup superblock (in an analogous way to what is suggested in an answer to a related question ), and I ended up with a bunch of folders in "lost+found" - fair enough. The problem is that some of these folders still give errors such as

ls: cannot access 'lost+found/#26128': Structure needs cleaning

... but if I now run again `e2fsck -f` (as suggested in other answers ), I get nothing strange

e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
EData: 191504/15269888 files (0.1% non-contiguous), 48227460/61049344 blocks

... and by the way, this only takes a couple of seconds, so it is clear that it's not actually check all the content of the disk. I tried to check for other options, and for instance tried `-E discard, but nothing changed: the check is still very quick, and some folders inside lost+found` still give the same errors. How can I fix the errors with those folders that say "Structure needs cleaning"? Note that the device is perfectly working from a hardware point of view.

I've got a bunch of photos that were on an SD card. Most of them are corrupted now, I cannot open them in any program (Gwenview, Gimp, on windows too...) But every file browser still can correctly make most of their thumbnail. for instance : I'm aware that the SD card is probably dying, so I already made a dd image of the sd card, on which I can work. So far, My searches led me to photorec which does not work in my case: it recovered 4 useless photos and as about two/third of my ~400 still have their thumbnail, I still hope to have most of those back. What Can I do? Any help appreciated :) PS: I'm on Kubuntu 20.04, I'm able to be root and I'm not afraid of the command line (but a graphical tool is still handy :D )

I have an external HDD drive formatted as Ext4 connected to my Raspberry Pi server. The server has been crashing lately and as a consequence the HDD is corrupt. When I plug to my computer and run sudo fsck /dev/sda I get the output

fsck from util-linux 2.40.2
e2fsck 1.47.0 (5-Feb-2023)
ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 
 or
    e2fsck -b 32768 

Found a gpt partition table in /dev/sda

I searched online and found [this answer](https://unix.stackexchange.com/a/628354) explaining how to find backup superblocks. However, when I run mke2fs I am not getting a straightforward output as I was expecting, neither on the device (sda) nor on the partitions (sda1 or sda2), and I don't want to proceed if I don't know what I'm doing. I'm afraid I will break something. These are the outputs: * sudo mke2fs -n /dev/sda

mke2fs 1.47.0 (5-Feb-2023)
    Found a gpt partition table in /dev/sda
    Proceed anyway? (y,N)

* sudo mke2fs -n /dev/sda2

mke2fs 1.47.0 (5-Feb-2023)
    /dev/sda2 contains a ext4 file system labelled 'data'
        last mounted on /srv/dev-disk-by-uuid-19f0b495-7b9d-4f67-ae8e-9f8933ff19eb on Sat Nov  2 13:36:52 2024
    Proceed anyway? (y,N)

* sudo mke2fs -n /dev/sda1

mke2fs 1.47.0 (5-Feb-2023)
    /dev/sda1 alignment is offset by 3072 bytes.
    This may result in very poor performance, (re)-partitioning suggested.
    /dev/sda1 contains a vfat file system labelled 'EFI'
    Proceed anyway? (y,N)

This makes clear to me that the problem is on partition sda1, but I'm still not confident to proceed with mke2fs. The way it asks Proceed anyway? makes me worry I will break something. Moreover, I don't know why the hard drive has 2 partitions nor what the EFI partition sda1 is (this is not a system drive, just an external data drive, although it has docker containers installed in it). Is mke2fs -n the right command for me here? If so, should I run it on sda or sda1? Should I repeat what I did and select Proceed anyway?

'Drive0' refers to the disk where system is installed. Can the **immutable** system be damaged (meaning can its filesystem that is mounted in read-only, not counting the files that are user accessible, be overwritten) if I run this command: sudo dd if=/dev/urandom of=/dev/drive0 Will the root user have enough privileges to access the drive directly and write zeroes to it or will the immutability of the system and other security related mechanisms such as OverlayFS prevent the root user from being able to write to the protected storage partition?

I have an embedded device which appends data to an ext4 file in data = ordered mode until its power is cut off. To my surprise, I've noticed that this occasionally results in zeros being appended to the file instead of the intended data. I thought this is impossible since data = ordered should guarantee that the data is written before the file size is updated. From the [docs](https://www.kernel.org/doc/Documentation/filesystems/ext4.txt#:~:text=data%3Dordered%09(*)%09All%20data%20are%20forced%20directly%20out%20to%20the%20main%20file%0A%09%09%09system%20prior%20to%20its%20metadata%20being%20committed%20to%20the%0A%09%09%09journal.) : > data=ordered > > All data are forced directly out to the main file system prior to its metadata being committed to the journal. Am I misunderstanding the meaning of "data" (file content) or "metadata" (file size) here, or what else is going on? Other parameters which may be relevant: - Writing is done using a FileOutputStream in Java v8. - OS is Ubuntu 18.04.6. --- Migrated from [StackOverflow](https://stackoverflow.com/questions/79109284/power-failure-while-appending-to-ext4-file-results-in-spurious-zeros) because apparently this question is not "programming-related" enough...

Disclaimer: I'm not very good at writing questions, and mine is a very specific scenario. So I'm just going to jump straight into the situation: I was browsing my filesystem, and somehow managed to download a tarfile containing two critical nodes (tarfile::todo/main and tarfile::todo/code) from my local FS, and write it to the same file. Once I realized what the downloader was doing, I quickly stopped it and checked the tarfile to find, to my dismay, that only a small chunk was left, and the rest was truncated off. I don't know why it didn't back up the existing file before writing something else to it, or why session.tar wasn't committed to my git repo, but now the whole thing's gone. I'm a very careful user, but when I mess up, it's absolutely catastrophic. After extundelete failed to recover the file, I browsed around here to find [this](https://unix.stackexchange.com/a/150423/54466) , which gives a method combining grep and dd to find and read the data directly from the hard drive. More context: * My /home/user directory is mounted on a separate drive: /dev/sdb3 * Today is day two trying to recover the files. * The output in the next paragraph was generated today. Once I did the grep/dd combo, I got [this output](https://unix.stackexchange.com/a/150423/54466) . How in the world do I use this to get my files back? I tried copying it into a .file.swp and recovering with vim -r in the hopes that it was a vim swapfile, but it isn't. I have never seen this format before, so I have no idea what any of it means. I would *really* like to get this data back. As I said before, it's critical. It's the veritable basket that has all of my eggs. Losing it would be a painful blow to productivity and organization.

I have a computer at home that I use as a file server, using Nextcloud. Every computer at home is Ubuntu/etx4. I've had multiple outages recently in the neighborhood, and I think the sudden shutdowns have messed up my external hard drive. It doesn't stay mounted, and folders seem to disappear and reappear on it. I checked the filesystem using Ubuntu's 'Disk' app and it said the disk is fine. However, running fsck /dev/sdb returns:

ext2fs_open2: Bad magic number in super-block
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sdb

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 
 or
    e2fsck -b 32768 

Found a gpt partition table in /dev/sdb

OK,so I ran e2fsck -b 32768 /dev/sdb:

e2fsck 1.46.5 (30-Dec-2021)
e2fsck: Bad magic number in super-block while trying to open /dev/sdb

I ran e2fsck -b 8193 and got the same result. Here are my questions: 1. What should be my next step? Testdisk? Something else? 2. This is a pretty new USB hard drive from Western Digital—a few months old. I could clone an image of the data, but I already have backups of this data elsewhere and could also just wipe the HD, start fresh, and move the data back over. But I don't know if it's safe to keep things backed up on this HD anymore. How do I determine whether the drive has hardware damage?

Twice in 6 months - on three Ext4 disks - one of them a Seagate that is about a year old - I have had catastrophic corruption of the filesystem. What seems to happen is that in many directories the list of directory entries is truncated, leaving only "." and "..", and much of the metadata structures get overwritten with straight 0xFFs. That is, a block will consist solely of the the byte 0xFF repeated 4096 times. I use Ubuntu Mate, 18.04.3 LTS, 4.15.0-72-generic #81-Ubuntu SMP. In the first instance about 6 months ago, I took two disks out of a machine that had suffered a power supply failure, mounted them in another machine without funning fsck on them. I did not write any data to the disks, but since I neglected to mount them read-only, metadata was probably written. Eventually I ran fsck on them both, and the above corruption happened. Specifically, the superblock, all backup superblocks, all copies of the GDTs, and large parts of the inode tables were overwritten with 0xFF bytes. By writing code to scan the disk for inode records outside of the proper tables and for directory files, I was able to cobble much of the filesystems back together. I estimate I recovered around half of the data on one disk, and 75% on the other. Well, I was pretty careless to mount the drives RW without running fsck on them, even though I did not overtly write data. I took this as a hard lesson to not be so careless. Curiously enough, I did find some of the missing metadata in the wrong places on the disk. This is how I was able to recover the superblock, and quite a few inode table entries. It appeared as if the system in its confusion wrote them to several data areas on the disk. Filtering them by checksums, I found 5-10 copies of some of the items in these scans. However, only a few days ago, similar corruption happened again (perhaps not as severe, I hope). I had a data disk with a single Ext4 partition on it, default options, in a machine that was having power-supply problems. The machine kept spontaneously turning off. The disk was a 4TB Seagate that was only a year old. I finally got a replacement power supply, installed it, booted it up, and found that the data disk was corrupted. I made several (5 ?) attempts to boot, each time getting dropped into emergency/safe mode. The annoying splash screen hid from me what was going on during the boot attempts. I finally realized that the boot was failing because my data disk was listed in fstab, but was not mountable. Removing it from fstab, the system booted ok. Looking at the disk, I found it corrupted. The superblock - exactly all of block 0 - was all 0xFFs. However, the backup suberblocks were fine, as were the GDTs. Of the inode entries in use, around 0.5% of them was also overwritten with straight 0xFFs - including the first inode block (inodes 1-11) (that is, 99.5% of them look fine). The 0xFF overwrite was always an integer number of blocks, on block boundaries. As I have begun to examine the filesystem in more detail, I have found nearly all of the inode entries of directory files were overwritten with the 0xFFs - a whole block at a time. So, although its only half a percent of the inodes in use, its the most inconvenient half of a percent to loose. Also, several of the higher-level directory files looked as though all of the entries (except ".", "..") had been deleted. That is, the rec_len field of the ".." entry was extended to reach to the end of the block, and a subsequent block of the directory file had the inode number of the first entry set to zero. So, it looks as if when I tried to boot, fsck was automatically run. For reasons I do not understand, it deleted all files in several directories, and then overwrote key metadata blocks with straight 0xFFs. It did this automatically - without asking for my approval for this "fix". I have several questions: 1. Could this be due to the filesystem journal getting corrupted, and then fsck going nuts trying reconcile? If not, what could cause this? 2. Is there some reason that Ext4 with journaling is actually more vulnerable to data loss - through this failure mode - than say, an older ext2 or ext3 filesystem? 3. How can I configure the next data disk filesystem to make as certain as possible this never happens again? I plan to recover what I can of the most recent data disk, and then set up an Ext4 filesystem with checksumming enabled. Also, I heard that there was an option to enable checksumming of the journal - is that advisable? Reliability is more important to me than performance - but I also don't have the funds to set up a large raid array. Is there a better filesystem than Ext4 for my purpose? 4. Can I set an option in Ubuntu Mate to limit what fsck will attempt to automatically do to "fix" my filesystem (on boot, or otherwise)? Would it be better for me to disable journaling? I don't want fsck automatically deleting files! In fact, I am seriously considering just deleting fsck and writing my own repair code - but there has got to be an easier way. I googled around online to see if other people have had this happen to them, and can not find any other cases. If this is so common of a failure mode to happen to me three times in six months, surely it must have affected many other people out there. However, I can not find any other descriptions of this problem on the web.