How do you synchronize files with partially failed download using rsync?
0
votes
1
answer
93
views
I'm trying to synchronize literally thousands of files of various sizes and I would like to have a 1:1 copy of the files. That means that already present files should be checked for their integrity and if there's a wrong checksum, the file needs to be overwritten. A so-called delta transfer is only necessary at this point because of the partially failed transfer.
Apparently my mount is kinda unstable and it fails after 300-400GB of transfer using
cp
or rsync
.
I did the following before this:
1. I mounted the storage, and did cp -r src dest
, it failed after like 300GB because the mount dropped and it errored out (don't have the error anymore apparently)
2. I mounted the storage again and did rsync -aP src dest
, it failed after like 400GB with rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
because the mount failed again. Considering the file size it probably overwrite most of the files.
3. I checked my kernel log and found nothing (sudo dmesg
)
I found a reconnect flag for my mount, but it would not be instant.
- There's a rsync flag named -c
which calculates the checksums, but does it do a so-called delta transfer too or do I need to add more flags?
How could I best fix this problem at hand?
# UPDATE 1
Correct me if I'm wrong but I think the issue at hand was that the storage had the owners and groups of different users and groups than in rsync. To elaborate: cp -r
copied the files and changed their ownership and group ownership to the user copying, whereas rsync seems to copy the file 1:1 with the same user and group ownership... That's probably why the transfer was overwriting old files...
Asked by Sir Muffington
(1306 rep)
Dec 30, 2023, 03:26 PM
Last activity: Dec 30, 2023, 04:24 PM
Last activity: Dec 30, 2023, 04:24 PM