After I rescue some disk, I copy the result (disk image of several TBs ) from local network data storage (LNDS) to target disk as a file. Usualy the filesystem of the target disk is NTFS. The LNDS is capable to pass data to me at the speed > 600MB/s (on 10Gbit network).
The target disk is capable to save data at sustain speed >150 MB/s. But the speed of copying file from the server to target disk filesystem is damned slower, etwa 60-80MB/s.
Today I was asked to prepare two copy. Well, I used the tee command to specify two targets:
dd if=/servermount/path/file.img bs=8192 |tee /localmount/target1/file.img >/localmount/target2/file.img
(
cp
cannot be forked with tee
) But what surprized me: the copy speed is about 40MB/s only.
I immediately started to analyse the trafic: The network is loading data at the sustained speed of 40MB/s, and both disks writes at max speed 160MB/s but less then 25% of the time.
I have looked the top
and found that the CPU spends most time with two mount.ntfs
processes - each takes >40% - and tee
which takes about 30% of CPU time.
I tried also a revers process: I started a samba server on my workstation, added the two targets as share, and sended the data from server to the two disks.
I was surprised again, because the common speed has grown to 66 - 69 MB/s and the mount.ntfs processes were spending just about 7% CPU time.
I cannot understand, why there was such great difference between push and pull over the net, and other way than it could be expected.
Not all bash command are suitable for piping with tee
. The dd
if targeted to /dev/null
gives the speed of 650-700MB/s. If I copy the physical disk to two other phys. disks (no metter if making clones or creating two disk images) the copy speed is over 120MB/s, and the same I get if the target is on the server (LNDS).
Can some body explain me, what I did wrong. Is there some copy command more suitable for copying large file from LNDS to two local disks?
Asked by schweik
(1440 rep)
Aug 23, 2021, 01:45 PM
Last activity: Aug 24, 2021, 09:49 AM
Last activity: Aug 24, 2021, 09:49 AM