Is there **an option in rsync to fetch all its metadata** about the destination tree, or a
--link-dest
tree,\
**from a cache file** instead of a full traversal of the filesystem?
##### Use case
I currently do daily backups of either 2 TB from a dedicated server, or from home PCs or smartphones, to a home (slow) disk plugged to a Raspberry Pi 3; or the the other way round (home PCs → dedicated server).
I use a rotating destination, so in essence my rsync command is:
rsync -az --delete --link-dest=../. :/ /backup/. \
&& mv /backup/. /backup/.
Inbetween, my backup directories are read-only, so when I reuse /backup/.
or /backup/.
I know **their contents have not changed since they where created by rsync
**.
##### Level 1: Read & dump
So I would like to **speed up the transfer by skipping the "building file list" phase** of rsync
, for the destination side (I'm not interested in the source, as 1. they're moving objects (else I wouldn't backup!) and 2. it mostly comes from SSDs).\
Of course I take responsibility for not modifying the backup, so the rsync-managed metadata dump stays in sync, else I'm OK rsync
giving up.
I imagine such an option (or a manual wrapper around rsync?) would:
* if detecting the dump (and it has all the info needed by this run of rsync
: checksums for a -c
, and so on): read it and send it instead of a full traversal
* just before the first write to the dest (so _not_ in -n
mode, nor on --link-dest
): truncate or unlink that dump\
because it isn't guaranteed in sync with the tree anymore
* at the end, dump the metadata corresponding to the data synced into the dest\
(probably incrementally written to a temp file during the transfer, so it's just a rename then)
##### Level 2: no, really, do trust me
OK I said I never modified the destinations.
… Well now let's amend it: I never modify the dest, _except I sometimes replace some files with an hardlink to a static file from another backup_, of course after ensured they have the same checksum (and I don't mind the other attributes changing: r--
vs rw-
, and so on);\
that happens when I detect that the same 300 MB video has been backed up twice, once from the smartphone that captured it, and once from the family PC where it has been copied to.
In this case I would be glad the dump _trust me that the file in the tree is as good as what the metadata cache says_.
Of course I'd like it not to stumble upon it (it should not have stored inodes, or store it but fallback to a conventional path traversal in case it doesn't exist anymore),
but it even shouldn't try to access the filesystem to "validate" the dump.
Asked by Guillaume Outters
(109 rep)
May 12, 2025, 03:19 AM
Last activity: Jun 26, 2025, 12:23 PM
Last activity: Jun 26, 2025, 12:23 PM