Add error correction to SquashFS images as part of a backup strategy
2
votes
0
answers
87
views
My backup strategy currently primarily consists of daily backups of all of my machines with Borg Backup, stored on different storage devices in different locations, following the 3-2-1 strategy. These file-level backups are the important ones that matter most to me. My question is *not* about these backups, I mention them for context.
Next to Borg Backup I also sporadically create full disk image backups with
dd
. After zero'ing the disk's free space (using zerofree
on ext4, dd if=/dev/zero
otherwise) I usually create a SquashFS image of the raw disk image (e.g. sda.img
becomes the only file in disks.sqfs
). This allows me to store the raw disk image in compressed form, while still allowing me to access the data without the need to decompress everything first.
These full disk image backups are stored on a single storage device (a NAS to be more precise), i.e. don't follow the 3-2-1 strategy. Creating a second copy of the data is out of scope, simply because they take too much space and because I consider investing into more storage a waste of money due to my Borg Backup backups. So, I'm fine with loosing these backups per-se, but I want to protect them a little better. Thus I'm thinking about adding some sort of error correction mechanism.
I read through a lot of resources and found that Reed–Solomon error correction seems to be the way to go. It adds some overhead to the data stored and provides safety in most, even though not all cases.
**My question is the following:** How do I do that in practice? What tools are available and how would I use them in my case? I found [this 10 years old Stack Exchange question](https://unix.stackexchange.com/questions/170652/is-it-possible-to-add-error-correction-codes-bch-rs-or-etc-to-a-single-file) listing a whole bunch of tools, but many of the projects are apparently dead. Plus, they don't seem to fit my needs:
Storing the data in compressed form and yet being able to access the data without the need to decompress it first is a must-have for me. So, unless there's another solution, I'm stuck with SquashFS. However, according to the resources I read, combining ECC with compression is hard: One apparently shouldn't calculate ECCs from compressed data, but from the original data, because ECC doesn't guarantee a 100% correction and even a single remaining corruption could yield all compressed data useless. However, calculating ECCs from original data and then compressing it wouldn't help either, because I might not be able to decompress the data due to the corruptions. So, apparently one needs software that does both at the same time: compression and ECC. Per ddrescue
I found that lzip
can actually do that by creating forward error correction (fec) files alongside compressing the data, but AFAIK I can't tell SquashFS to create these files.
So, I'm kinda stuck with this chicken-and-egg problem... How can I combine SquashFS with ECC, or is there an alternative to SquashFS that allows this?
Any suggestions?
Asked by PhrozenByte
(21 rep)
Apr 1, 2025, 07:26 PM
Last activity: Apr 1, 2025, 07:30 PM
Last activity: Apr 1, 2025, 07:30 PM