Sample Header Ad - 728x90

How to shard file into n-out-of-m redundancy (erasure code e.g. kind of reed solomon)?

8 votes
2 answers
1144 views
How to shard file file into *m* files, so it can be recovered with any *n* of them ? It looks like [Erasure Code](https://en.wikipedia.org/wiki/Erasure_code) , preferably "optimal erasure codes". (Example of another application and proposed programming library: "You need erasure code" https://stackoverflow.com/a/28932095 ). It's like [Reed-Solomon error correction](https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) (something more flexible than [RAID6](https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6)) style of redundancy. Early findings: I've found rsbep so far and [some modification](https://www.thanassis.space/rsbep.html) , but they rather seem to be designed toward different use case. I've also found reed-solomon from Linux Kernel ported to userspace [here](https://github.com/tierney/reed-solomon) , but it's not the tool for my described purpose. Example for 3-out-of-6 level of redundancy: split_with_redundancy -n 3 -m 6 input.dat Producing input.dat.0..5, so any tree of those files are sufficient for recovery: recover_using_redundancy intput.dat.{0,2,4} I do not care for errors within given file, i.e. I do not need [Forward Error Correction](https://en.wikipedia.org/wiki/Forward_error_correction) . I assume that I can rely on having n-out-of-m redundant parts fully correct.
Asked by Grzegorz Wierzowiecki (14740 rep)
Apr 20, 2016, 07:28 PM
Last activity: Sep 16, 2020, 07:15 PM