Sample Header Ad - 728x90

Smarter filetransfers than rsync?

10 votes
3 answers
10816 views
I have a large file (2-3 GB, binary, undocumented format) that I use on two different computers (normally I use it on a desktop system but when I travel I put it on my laptop). I use rsync to transfer this file back and forth. I make small updates to this file from time to time, changing less than 100 kB. This happens on both systems. The problem with rsync as I understand it is that if it think a file has changed between source and destination it transfers the complete file. In my situation it feels like a big waste of time when just a small part of a file has changes. I envisage a protocol where the transfer agents on source and destination first checksums the whole file and then compare the result. When they realise that the checksum for the whole file is different, they split the file into two parts, A and B and checksum them separately. Aha, B is identical on both machines, let's ignore that half. Now it splits A into A1 and A2. Ok, only A2 has changed. Split A2 into A2I and A2II and compare etc. Do this recursively until it has found e.g., three parts that are 1 MB each that differs between source and destination and then transfer just these parts and insert them in the right position at the destination file. Today with fast SSDs and multicore CPUs such parallelisation should be very efficient. So my question is, are there any tools that works like this (or in another manner I couldn't imagine but with similar result) available today? A request for clarification has been posted. I mostly use Mac so the filesystem is HFS+. Typically I start rsync like this rsync -av --delete --progress --stats - in this cases I sometimes use SSH and sometimes rsyncd. When I use rsyncd I start it like this rsync --daemon --verbose --no-detach. Second clarification: I ask for either a tool that just transfers the delta for a file that exists in two locations with small changes and/or if rsync really offers this. My experience with rsync is that it transfers the files in full (but now there is an answer that explains this: rsync needs an rsync server to be able to transfer just the deltas, otherwise (e.g., using ssh-shell) it transfers the whole file however much has changed).
Asked by d-b (2047 rep)
Jan 25, 2015, 04:37 PM
Last activity: Jan 15, 2024, 10:00 AM