Problem

My Armbian based Orange Pi webserver crashed (probably because of a power loss). I thought it would be fine since ext4 is more resilient and it usually has been in the past, but for some reason, this time, it didn't reboot, it hung. When I checked, it seems the "drive" (actually a memory-card) **"does not have a valid filesystem"**. I pulled the card and made a backup image (amounting to ~4.5GB out of the 32GB space on the card) and examined it in a hex-editor and various programs. I also burned an Armbian ISO to an identical memory-card to compare metadata values. I do have a copy of the system from a couple of years ago on a different (non-identical card), but the server has changed a fair bit since then. Hopefully it can at least provide some more information to compare.

Observations

I've noticed several issues: * Most programs are unable to detect the card as containing a filesystem. No, fsck, testdisk, etc. don't work, they complain about a bad magic number, or wrong fs type, bad option, bad superblock, or missing codepage, or filesystem seems damaged, or bad relative sector, or other blocking errors. * The superblock is damaged (and I couldn't find backups); it looks like most of the entries are okay, but the following definitely have unusual/invalid values. I compared the values of the broken system with the new one, leaving out values like partition size or mount count which will naturally differ. Expected values with a question mark are ones which I'm not sure are specific to the drive or not, that is, I don't know if they're incorrect. |Field|Actual value|Expected value| |-|-:|-:| |Blocks per group|0|32768| |Fragments per group|0|32768| |inodes per group|5680|? 8160| |Maximal mount count|0x7777|0xffff| |Magic signature|0x6753|0xef53| |File system state|🤨 ‽‽‽ 0x03|0x00 or 0x01| |First inode number|3|? 11| |Compatible features|0x34|? 0x3c| |Incompat features|0x246|? 0x242| |R0 comp features|0x63|? 0x7b| * Searching the drive with a hex-editor, I'm able to find various directories and files, but examining it with some programs seems to indicate that at least some inodes seem to be wiped/zeroed out. * If I modify some of the superblock values (specifically the …per group values, magic signature, and file system state, then programs can see it, but with the following problems: * They still complain about things like it being an invalid filesystem or partition type * Some programs can list the root directory, but only show a few subdirectories (bin, boot, dev, lib, lost+found) * Contents of displayed directories are incorrect (eg only a few Python files in /boot) * Other subdirectories (etc, mnt, …, usr, var) are shown as broken/unknown files instead of directories * At least one subdirectory (home) isn't listed at all—though that might be because home is a symlink to somewhere in one of the broken directories 🤔 * One program's validation routine shows the following problems: * Integrity status: **invalid** * inodes count is valid: **no** * SuperSector copy exists and matched: **no** (is that supposed to be superblock? 🤔) * Most programs are showing the root-directory as ' instead of /. 🤨 * The damaged card has ~300MB of unallocated space after the ~29GB partition. That may or may not have been like that before the corruption. I distinctly recall a Pi setup tutorial saying to run a command to expand the partition to the full size of the drive/card after burning the ISO and booting the first time, but I just can't seem to find the tutorial or command 🤷. When I burned the new card and expanded the partition, it used the the whole card. It does seem like the old version on the old card also has a ~290mb unallocated chunk at the end. * Sometimes I get the sense that something was deleting everything on the card, but that doesn't make sense because it wasn't a virus that crashed it, it was a faulty power-supply. 🤨

Resources

Here are some resources I have available at my disposal: * Original corrupted card with minimal modifications (which I can reverse) * Cloned image of corrupted card * Previous version of the system from 2019 before it was transferred to a new card (though I can't for the life of me remember if I cloned it to the new card and then updated it or just burned a new ISO from scratch and transferred some files 🤔—I know I started with Jesse, updated to Stretch somehow, and was using Buster before the crash) * Another identical memory-card that (should) have identical parameters (eg sector count) * Free or trial versions of as many data-recovery and/or forensic programs as I could find (I'd consider purchasing a license to something if it's not too expensive and it'll definitely help restore the server) * A WordPress SQL dump from two months before the crash (but it doesn't contain external files like graphics files 🤦) * Backups of the etc, home, var/lib/mysql, and var/www directories from a couple of years ago—I could swear I made a more recent backup of at least etc but can't seem to find it anywhere 😕 * ISOs of various versions of Armbian * Various screenshots * Unfortunately, I'm not too knowledgeable about ext* (yet; I'm reading up on it now), but I am very familiar with FAT* and have done a lot of data-recovery with that over the years, so hopefully my skills can transfer to this filesystem 🤞 * I'm happy to provide any other information or answer any other questions needed

Desire/Requirements/Question

Ideally, I'd like to be able to modify a couple of values to fix the filesystem so that it's all fixed like nothing happened, but it looks like that's not going to happen. As a last resort, I can restore the image from late 2019 and try to manually restore/re-inject the other individual backups, at the risk of possibly losing a couple of years of work. At a minimum, I need to recover: * At least a few things from etc (mostly text files) * Any new files in root/home since last version (mostly text files) * Any new files in var/www since the last version (mostly graphics files—WITH FILENAMES) * Ideally var/lib/mysql (binary files) to avoid losing two months since last SQL dump * List of installed packages (where does Linux keep the list? please be a text-file🤞) Remeber, this is offline, so I can't run a command to dump it (though I swear I did recently, but again, I can't seem to find it 🤦) (You know, looking at it like this, it doesn't seem so disastrous anymore; it feels like time and work but seems doable. 😀)

Question

**Is there a way/program that will let me rebuild the directory tree or inode chains? That is, some way to look for directories and files (eg a string/byte search), then fill in the sector/cluster/inode of the start of the item, then have the program rebuild the inode chain based on that and the length of the file/directory?** The card was at least 80% empty and there wasn't too much delete/write activity, so most of the files and directories should be unfragmented, so it should be sufficient for most of them to just know the start and length to be able to restore the directory or copy the whole file.

Note

A regular content-based file-recovery scan isn't going to be of much use since it would just dump a ton of files with no filenames or directory structure, which is useless here since many of the files will be similar and/or binary, and thus very difficult to rename or locate their appropriate directories.

Tangential question

How does the system know if the partition is ext2/ext3/ext4 if they all use the same partition identifier of 83/0x53? I couldn't find another field to identify it it. 🤔