Does a streaming version of
split
exist somewhere in Linux?
I'm trying to back up a large amount of data via SSH, but SSH's single threaded encryption is limiting the transfer. This machine doesn't have hardware AES support so I'm using ChaCha encryption, but the cpu is still not keeping up with the network.
So I thought I could solve this by splitting the data stream in two, and sending each over a separate SSH connection, and then merging the streams together at the destination. That way the encryption load can be shared over multiple cpu cores. This looked like a general enough idea that it should already exist, but I can't find it.
*edit*: for some numbers, I'm backing up data from an old computer, a few hundred GB over a gigiabit wired network. I'm copying an image from a partition, as that is faster than doing individual file access on the spinning rust drive, so technically it is random access data, but it is too large to treat it as such. I tried compressing it, but that doesn't help a lot. The data isn't very compressible.
So what I'm looking for is a split
(and corresponding merge
) that will split a stream of binary data into multiple streams (probably splitting by fixed chunks).
Asked by JanKanis
(1421 rep)
Mar 28, 2021, 04:29 PM
Last activity: Oct 12, 2023, 12:28 PM
Last activity: Oct 12, 2023, 12:28 PM