Sample Header Ad - 728x90

Avoid pipeline stalls from write caching to usb drive

2 votes
0 answers
71 views
I have the following process I use for backups:
btrfs send snapshots/home-2016-06-04 |
zstd --verbose -T4 - |
gpg --batch --passphrase-file /tmp/secret --compress-algo none --symmetric - >
/mnt/usbdrive/packup.zst.gpg
But I find that while it runs very quick at 100% CPU for a while, writes to the usb drive get buffered and then the whole pipeline suddenly stalls while it syncs to the drive. It then alternates between these two states. What I would like to achieve is for zstd --adapt -T1 to work properly. According to the man-page > zstd will dynamically adapt compression level to perceived I/O conditions. This should give the optimal throughput so that the write is at the maximum (40 MB/s - tested with dd) to the thumb drive, and so that zstd will not fill up it's buffer. I suspect I can achieve this by adding dd as the last stage of the pipeline; I've tried this with direct fsyncdata and sync options to oflag, to no avail. Is there some way to achieve this? Am I right in assuming the problem is with the write caching?
Asked by Luciano (141 rep)
May 20, 2019, 02:42 PM