Avoid pipeline stalls from write caching to usb drive
2
votes
0
answers
71
views
I have the following process I use for backups:
btrfs send snapshots/home-2016-06-04 |
zstd --verbose -T4 - |
gpg --batch --passphrase-file /tmp/secret --compress-algo none --symmetric - >
/mnt/usbdrive/packup.zst.gpg
But I find that while it runs very quick at 100% CPU for a while, writes to the usb drive get buffered and then the whole pipeline suddenly stalls while it syncs to the drive. It then alternates between these two states.
What I would like to achieve is for zstd --adapt -T1
to work properly. According to the man-page
> zstd will dynamically adapt compression level to perceived I/O conditions.
This should give the optimal throughput so that the write is at the maximum (40 MB/s - tested with dd) to the thumb drive, and so that zstd
will not fill up it's buffer.
I suspect I can achieve this by adding dd
as the last stage of the pipeline; I've tried this with direct
fsyncdata
and sync
options to oflag
, to no avail.
Is there some way to achieve this? Am I right in assuming the problem is with the write caching?
Asked by Luciano
(141 rep)
May 20, 2019, 02:42 PM