Sample Header Ad - 728x90

Exclude from rsync log file or output non-updated files

2 votes
1 answer
621 views
For rsyncs that involve a large number of potentially transferred files, I mean to have a "clean" log, with only what was transferred. I worked with stdout, since: 1) I did not know how to configure the log output (if that is even possible). 2) I also want this to work with --dry-run (which forced me to work with something like this , since). 3) I mean to write scripts useful for transferring Linux-Linux, and Linux-msys2 (and viceversa) at least. Here, options for permissions (-p), owner (-o) and group (-g) may complicate things. I combined command rsync --recursive --perms --group --owner --times --one-file-system --sparse --stats --progress --update --out-format='%t %p %i %n %M %l' --delete --dry-run --include="*" --exclude="*" --log-file=rsync.log ./ ../trg with ... | grep -E -v ' \.[fdLDS][\.c][\.s][\.t][\.p][\.o][\.g][\.u][\.a][\.x]' following the description here . The first dot matches any item that is not being updated. The second character matches any item type. The remaining characters match any condition on the attributes (change/no change) Flag -v reverts matching. This presumably matches any item that is being updated, and only that. Items not updated, but with changed timestamp, would not be matched. I finally combine this with tee, as in rsync --recursive --perms --group --owner --times --one-file-system --sparse --stats --progress --update --out-format='%t %p %i %n %M %l' --delete --dry-run --include="*" --exclude="*" --log-file=rsync.log ./ ../trg | grep -E -v ' \.[fdLDS][\.c][\.s][\.t][\.p][\.o][\.g][\.u][\.a][\.x]' | tee rsync.2.log which produces the rsync log file rsync.log, and my "custom" log file rsync.2.log. Even if it seems to work, the 3 "weak" points of my approach are the following: 1. I am not sure there are cases I am missing, where this approach might fail (by "excessive" reporting or filtering files). 2. I would like to handle this natively via rsync itself. Even better, getting the output in the rsync log file (not only its output), rather than using the brute force grep approach. Note that using --itemize-changes I get in stdout some files with e.g. status .f...p....., which should be filtered (as they are not updated, first character is a dot). 3. Using grep I cannot see the partial progress. I am not sure if for very large output this can be a major problem. **Related**: 1. https://unix.stackexchange.com/questions/203848/how-to-make-rsync-to-log-deleted-files 2. https://unix.stackexchange.com/questions/508331/how-to-make-rsync-display-only-files-being-copied-not-all-the-folders-which-are 3. https://unix.stackexchange.com/questions/44372/logging-only-transferred-files-with-rsync 4. https://superuser.com/questions/1002074/linux-command-line-to-create-a-log-file-for-rsync 5. https://serverfault.com/questions/401210/rsync-report-only-uploaded-files **Note**: The subtleties in this question (see also answer) make it slightly different from other similar questions posted/linked. Whether this warrants a separate question is a blurred line, and subjective. I personally find it useful to leave it as it is now.
Asked by sancho.s ReinstateMonicaCellio (2998 rep)
Jan 24, 2022, 09:09 PM
Last activity: Jan 28, 2022, 03:24 PM