How to shard file into n-out-of-m redundancy (erasure code e.g. kind of reed solomon)?
8
votes
2
answers
1144
views
How to shard file
file
into *m* files, so it can be recovered with any *n* of them ?
It looks like [Erasure Code](https://en.wikipedia.org/wiki/Erasure_code) , preferably "optimal erasure codes". (Example of another application and proposed programming library: "You need erasure code" https://stackoverflow.com/a/28932095 ).
It's like [Reed-Solomon error correction](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) (something more flexible than [RAID6](https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6)) style of redundancy.
Early findings: I've found rsbep
so far and [some modification](https://www.thanassis.space/rsbep.html) , but they rather seem to be designed toward different use case. I've also found reed-solomon from Linux Kernel ported to userspace [here](https://github.com/tierney/reed-solomon) , but it's not the tool for my described purpose.
Example for 3-out-of-6 level of redundancy:
split_with_redundancy -n 3 -m 6 input.dat
Producing input.dat.0..5
, so any tree of those files are sufficient for recovery:
recover_using_redundancy intput.dat.{0,2,4}
I do not care for errors within given file, i.e. I do not need [Forward Error Correction](https://en.wikipedia.org/wiki/Forward_error_correction) . I assume that I can rely on having n-out-of-m redundant parts fully correct.
Asked by Grzegorz Wierzowiecki
(14740 rep)
Apr 20, 2016, 07:28 PM
Last activity: Sep 16, 2020, 07:15 PM
Last activity: Sep 16, 2020, 07:15 PM