Sample Header Ad - 728x90

How do you synchronize files with partially failed download using rsync?

0 votes
1 answer
93 views
I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer. Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using cp or rsync. I did the following before this: 1. I mounted the storage, and did cp -r src dest, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently) 2. I mounted the storage again and did rsync -aP src dest, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] because the mount failed again. Considering the file size it probably overwrite most of the files. 3. I checked my kernel log and found nothing (sudo dmesg) I found a reconnect flag for my mount, but it would not be instant. - There's a rsync flag named -c which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags? How could I best fix this problem at hand? # UPDATE 1 Correct me if I'm wrong but I think the issue at hand was that the storage had the owners and groups of different users and groups than in rsync. To elaborate: cp -r copied the files and changed their ownership and group ownership to the user copying, whereas rsync seems to copy the file 1:1 with the same user and group ownership... That's probably why the transfer was overwriting old files...
Asked by Sir Muffington (1306 rep)
Dec 30, 2023, 03:26 PM
Last activity: Dec 30, 2023, 04:24 PM