I use unison to sync two folders that are in separate computers (via unison command with ssh). If for convenience I happen to change the name of the parent folder, unison start to sync the folders as if it were the first time I sync them, taking a lot of time (because the two folders are quite large). Here is the message I get: Looking for changes Warning: No archive files were found for these roots, whose canonical names are: /path/to/my/folder1 /path/to/my/folder2 This can happen either because this is the first time you have synchronized these roots, or because you have upgraded Unison to a new version with a different archive format. Update detection may take a while on this run if the replicas are large. Unison will assume that the 'last synchronized state' of both replicas was completely empty. This means that any files that are different will be reported as conflicts, and any files that exist only on one replica will be judged as new and propagated to the other replica. If the two replicas are identical, then no changes will be reported. Is there a way to tell unison that these folders have previously been synced? Perhaps changing the path somewhere and recovering the last unison file? Similarly, maybe a related question: if I copy one of my folders to a new computer and therefore they are already in that moment identical, could I tell unison that they are identical to skip the update detection process?

I am trying to synchronize files between two servers: server1 and server2.  I created a user and gave him permission to read, write and execute (rwx) on /var/www, and then I added the following Unison configuration file: root=//var/www/html/ root=ssh://user@server2//var/www/html/environments/simreg/ log = true ignore=Path cache ignore=Path vendor ignore=Path logs auto=true batch=true repeat=watch #confirmbigdel=true confirmbigdel=false fastcheck=true prefer=newer silent=false times=true owner=true server = true confirmmerge = true backupdir = /var/www/html/environments/backup ui = text But when I try to run Unison, it is unable to synchronize.  The debug is not useful, and I do not understand what I am doing wrong. Here is the debug print: user@server2:~$ unison -debug all [startup] Preferences: ui = text host = server = true prefsdocs = false doc = version = false silent = false dumbtty = false testserver = false showprev = false selftest = false confirmmerge = true retry = 0 repeat = watch contactquietly = false key = label = expert = false height = 15 auto = true maxthreads = 0 prefer = newer force = sortnewfirst = false sortbysize = false keeptempfilesaftermerge = false diff = diff -u CURRENT2 CURRENT1 backupdir = /var/www/html/environments/backup maxbackups = 2 backups = false backupsuffix = backupprefix = .bak.$VERSION. backuploc = central copymax = 1 copyquoterem = default copythreshold = -1 copyprogrest = rsync --partial --append-verify --compress copyprog = rsync --partial --inplace --compress rsync = true fastcheck = true ignorelocks = false dumparchives = false ignorearchives = false showarchive = false rootsName = fat = false allHostsAreRunningWindows = false someHostIsRunningWindows = false ignore = Path logs ignore = Path vendor ignore = Path cache confirmbigdel = false batch = true root = ssh://user@server2//var/www/html/ root = //var/www/html/ killserver = false halfduplex = false stream = true addversionno = false servercmd = sshargs = rshargs = rshcmd = rsh sshcmd = ssh xferbycopying = true sshversion = ignoreinodenumbers = false links-aux = true links = default times = true group = false owner = true numericids = false dontchmod = false perms = 1023 rsrc-aux = false rsrc = default maxerrors = 1 unicodeCS = false unicodeEnc = false unicode = default someHostIsInsensitive = false ignorecase = default timers = false terse = false logfile = /var/www/html/.unison/unison.log log = true debugtimes = false debug = all addprefsto = Contacting server... [remote] Shell connection: ssh (ssh, -l, dev_user, server2, -e, none, unison, -server) user@server2's password: [globals] Checking path '' for expansions Connected [//server1//var/www/html/ -> //server2//var/www/html/] [startup] Roots: //var/www/html/ ssh://user@server2//var/www/html/ i.e. //var/www/html/ ssh://user@server2//var/www/html/ i.e. (in canonical order) /var/www/html/ //server1//var/www/html/ [props] Setting permission mask to 1777 (1777 and 7777) [stasher] initBackupsLocal [stasher] d = / [stasher] Prefix and suffix regexps for backup filenames have been updated [server: stasher] initBackupsLocal [server: stasher] d = / [server: stasher] Prefix and suffix regexps for backup filenames have been updated [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... [ui] Nothing changed: sleeping for 10 seconds... But files are located on both servers.

I have a setup in which I sync files between two Linux servers using Unison over an SSH connection. This is implemented by running the following command via cron: unison -auto -batch PROFILE Changes to the fileset happen almost exclusively on one side (the other system is an off-site replica). They mostly take place through another Unison sync run (triggered manually) with a client machine, and can be from a few hours to a few weeks apart. Thus, conflicts between the two machine are not much of a concern in practice, and a delay of up to 24 hours for changes to get propagated to the other side is acceptable. Reasons I am running Unison as a cron job rather than with -repeat (presumably as a systemd service) are: * Predictable sync times, as the cron job is scheduled at a time when I am not expecting any manual sync operations from the third machine (whereas, say, -repeat 86400 would drift by the duration of the sync operation). * Changes mostly happen on server A, while the server-to-server sync job is triggered by server B (is it is easier network-wise if server B initiates the connection). Thus, as I understand, -repeat watch would not pick up most of the changes and even with -repeat watch+TIME, I’d be relying on TIME almost exclusively (correct me if I missed something). When changes do happen, they are usually low in volume. Occasionally, however, the data volume to be transferred is such that a single Unison run lasts several times as long as the interval between two Unison cron jobs (bandwidth between the systems is somewhat constrained). That would mean one Unison process is still running when cron launches the next one on the same set of files. I take it that Unison has lock mechanisms in place, which presumably prevent the “new” process from messing with anything the “old” one is working on (but correct me if I’m wring or have missed something). But I’m wondering what the second Unison process would do in that case – I have observed it does not exit but stays around. Does that mean the second process would wait for the first to finish and only then start synchronization (which would then only include files that changed while the first sync was in progress and therefore failed to sync on the first run)? Is it safe to launch a second Unison process while another one is still running on the same profile? (If not, what is a recommended way to prevent two concurrent Unison instances if, and only if, they are at risk of interfering with each other?) What about the resource overhead of unison -repeat wait+TIME vs. occasionally having multiple Unison instances queued up, one running and the others waiting for it to finish?

I have a file server running Raspbian 9, and use Unison over SSH to sync my home dir between these two. This has worked well for the two years I was on Ubuntu 18.04, but after upgrading to Ubuntu 20.04, syncing over SSH aborts with an unmarshaling error, claiming the two Unison versions (local and remote) were built with different OCaml versions. Unison version is 2.48 on all systems involved (pre and post upgrade). What gives?

I have a file server running Raspbian and a workstation running Ubuntu MATE 16.04. I use Unison to keep my home directory in sync. I’ve set it up to use a SSH connection. The workstation runs Unison 2.48; the server has multiple versions (including 2.48). I’ve successfully propagated a couple of changes from the workstation to the server that way. Now I’ve changed a file on the server. Unison looks for changes and correctly suggests updating the local copy with the file from the server. However, when I click **Go**, I get the following error: Fatal error Lost connection with the server. Unison settings are as follows: # Unison preferences label = Home dir root = /home/me root = ssh://me@fileserver//srv/file/me sshargs = -C addversionno = true dontchmod = true perms = 0 path = Documents path = Pictures General configuration problems (missing Unison version on server, wrong path etc.) are unlikely as changes are being detected and I have synchronized data various times (always workstation to server). It’s just the server-to-workstation scenario that fails. I decided to “sacrifice” my edits to the new file and chose to replace the server version with the outdated local one. Surprisingly, that sync run completed successfully—everything works unless there are changes on the server side. What’s happening here?

I have two machines. One with unison v. 2.40 (Debian Jessie) and a new one (Debian Stretch) where I installed package unison-all, which brought in unison versions 2.32, 2.40 and 2.48. However, I was unable to make my 2.40 work against the new machine, since it only presented v. 2.48. No matter what I do, I get: expected "Unison 2.40\n" but received "Unison 2.48\n\000\000\000\000\017", I worked around the issue by removing unison-all and installing v. 2.40 explicitly, but I wonder what to do if I ever want to use two concurrent versions of unison on this machine.

I want to create a sync system between newly available Proton Drive and my local filesystem. However, since it's very new, it's only barely supported by rclone (I'm lucky to be on openSUSE Tumbleweed because rclone in my other distros don't support it), and it doesn't have a sync client for Linux yet. I've considered 3 options: 1. mount the whole drive using rclone mount and work always online, but that's slow and sluggish and requires constant Internet connection; 2. mount drive with rclone mount and sync the folders I need using the tool [unison](https://github.com/bcpierce00/unison) . But my testing shows that it can take a very long time for the software to browse for changes... Even though when the scanning is done it's pretty fast for the correct/copy; 3. use rclone bisync. But it's advertized to be in testing and not very stable, and anyways I have the same problem that I have with unison, it's very slow to browse for changes... So if anyone has a better idea how to do that ("that" being setting up a sync system where when I change/add/remove something either on the computer or the drive, the change propagates to the other side immediately — or at least, fast, like, under a minute delay), please help!

rclone v1.64.2
- os/version: opensuse-tumbleweed (64 bit)
- os/kernel: 6.5.9-1-default (x86_64)
- os/type: linux
- os/arch: amd64
- go/versiorclone v1.64.2
- os/version: opensuse-tumbleweed (64 bit)
- os/kernel: 6.5.9-1-default (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: dynamic
- go/tags: none
n: go1.21.3
- go/linking: dynamic
- go/tags: none

That's the latest version of rclone. I'm using Proton Drive's cloud storage solution. rclone config redacted gives:

[proton]
type = protondrive
username = my.username
password = XXX
mailbox_password = XXX
client_uid = XXX
client_access_token = XXX
client_refresh_token = XXX
client_salted_key_pass = XXX
replace_existing_draft = true
### Double check the config for sensitive info before posting publicly

I have unison running under systemd to keep my local and remote NASs in sync. The systemd config is thus: [Unit] Description=NAS Unision service After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=nas ExecStart=/usr/bin/unison borg [Install] WantedBy=multi-user.target And in ~nas/.unison/borg.prf I have configure unison to run with: # Keep repeating every 15 mins. repeat=900 # Log to the file which will be rotated by logrotate. log=true logfile=/var/log/unison-borg.log logrotate is configure to rotate the log once per day: # Rotation of the Unision logs /var/log/unison-borg.log { dayly rotate 7 compress delaycompress missingok notifempty create 644 nas nas } My issue is that unison doesn't appear to open the log file with each repeat and thus when logrotate rotates the log file unsion-borg.log.1 gets the updates. The man file for unison states that "Sending SIGUSR1 will close the logfile; the logfile will be re-opened (and created, if needed) automatically, to allow for log rotation." How should I do this in systemd?

When I merge two folders with rsync, if files of the same name exist in the "target" directory, those will be kept unconditionally. I can use unison which is a bit more advanced, but as far as I know it only looks at file creation time and picks the newer one. In my case, files under the same path with the same name are indeed semantically the same, but they might have different sizes. I want to merge based on file size and not creation time. Is there a utility that supports this case or do I have to write a script for that? Unfortunately, I can't give an estimate of how many differently-sized files will be encountered. Might be anywhere between a small double-digit number to somewhere in the 10'000s.

I have just rebuilt my file server after it suffered disk corruption. When I tried to re-sync data using Unison, I got the error message > Lost connection with the server I retried from the command line and got: ~$ unison someprofile -ui text Contacting server... Unison failed: Fatal error: Error in creating unison directory /home/user/.unison: File exists [mkdir(/home/user/.unison)] Fatal error: Lost connection with the server I can open an SSH connection to the server. Both client and server run the same Unison version. What’s up here?

[ EDIT: Answered my own question. Used unison and some hacky post-processing ( copy-pasted the log output of unison, tweaked it in my text-editor with multi-selection editing, then did some shell script processing on that (fishshell) ) ((I've got so much more disaster recovery stuff to slog through, so I guess I'm done with *this* for now...)) ] --- So I have an SSD that used to be inside another computer, and I put it in one of those little SATA-to-USB enclosure/adapter shell things, mounted it as an external data drive, and used rsync -aAX to copy the boot partition over into a dir on this computer, for a backup. But then, after some other events that *probably* didn't change the contents of the original boot partition, I made a second backup. So now I have two dirs on this computer, which I think are *probably* two copies of exactly the same backup, but I want to make sure. --- So my question is: **What is the best way to compare/diff these two big backup directories?** Summary of things I have considered/tried but had problems with or am unsure about: - diff itself - rsync "dry run" trick - unison [(just thought of, but it hasn't finished running yet, due to the large size of the backups and the slow speed of my old hardware.)] Is one of these essentially a good option? If so, any corrections to the details of how I should go about using it? Or are there any separate, additional options I should know about..? --- # Details of my attempts and the results/problems: --- ## diff The obvious way of doing it for "*normal*" directories would be like: $ diff -r dir_A dir_B ( or maybe $ diff -r --no-dereference dir_A dir_B? I don't know; I honestly don't properly understand the function of --no-dereference -- it's just something I've found got me the results I wanted in *vaguely* similar situations in the past. ) However, the problem with using diff is that these dirs are of course very unusually large, and full of "weird" files from the bootable system (eg "character special files" and "block special file" etc). --- ## rsync So it occurred to me to just use rsync again between them, doing a "dry run" and seeing if it reported any changes it would make. Like: $ sudo rsyncy -n -aAX dir_A dir_B --log-file=log_file However, it then occurred to me - "what if there were *new* files in dir_B?" - "would rsyncy necessarily always report that?" So I guessed you would have to check both: $ sudo rsyncy -n -aAX --delete dir_A dir_B --log-file='log_file[A-to-B]' and $ sudo rsyncy -n -aAX --delete dir_B dir_A --log-file='log_file[B-to-A]' which is starting to feel a little suspiciously like maybe that isn't really the right tool for the job after all...? The log files I got read: A-to-B #=>

2023/07/21 01:43:04  building file list  
2023/07/21 02:12:24  sent 80.58M bytes  received 292.46K bytes  45.93K bytes/sec  
2023/07/21 02:12:24  total size is 229.29G  speedup is 2,835.29 (DRY RUN)

B-to-A

2023/07/21 01:41:58  building file list  
2023/07/21 02:12:15  sent 80.58M bytes  received 292.50K bytes  44.49K bytes/sec  
2023/07/21 02:12:15  total size is 229.29G  speedup is 2,835.29 (DRY RUN)

which are (ignoring the timestamps and speed information), annoyingly *ALMOST* the exact same: both sent 80.58M bytes but *tiny* different received: received 292.46K bytes vs received 292.50K bytes So yeah, again, I'm feeling doubtful this rsync trick is really the right tool for the job...? Maybe the correct answer really *is* like: " Just be patient and let diff run for ages to process the two huge directories. (You can just ignore all the error messages about special file etc.) " ?? --- ## unison [not sure yet?]

I have two machines that one runs Ubuntu 18.04 and the other one Ubuntu 20.04. I want to use Unison to sync files between them but there are some issues. Firstly, because I sync with another machine too, I compiled from source Unison 2.48.3. Then in the Ubuntu 18.04, I installed from the Ubuntu software Unison 2.48.4 and so I did with Ubuntu 20.04. After setting them up properly, from the command line of Ubuntu 18.04 I run

-2.48.4-gtk

and try to sync. Although everything works fine, i.e. it finds the machine and the directory then I get the message:

error during unmarshaling (input value:ill-formed message) possibly because client and serer have been compiled with different versions of OCaml compiler

. In the 20.04, I checked the version of OCaml but apparently it returned nothing (even though I had installed unison from app store). On the 18.04 the version is 4.05.0. Any ideas on what I can do?

I've been successfully using **rsync** to synchronize my home directories between my laptop and my netbook (both using ext4 with Linux). My only problem with it is that every now and then I like to completely reorganize certain directories, and **rsync** sees this as deletions and creation of new files, which makes it very slow and inefficient. Does anybody know of a good piece of software that handles file moves well? I've found that **unison** should be capable of this, but it doesn't seem to work in practice. I tested **unison** by synchronizing two local directories, each with a large file in it, and it still detected my move as a deletion+creation and was in effect even slower than **rsync**. Moreover, if I remember correctly from my looking at **unison**'s source code a couple of months ago, it tried to do something like doing sha sums—and I don't want my ideal solution to do sha sums—that's too slow on large files/directories. I'd like something that notices things like: " The last source counterpart of this file seems to have been deleted and there's a new file elsewhere in the source directory tree that has the same mdate, size, and inode number, so I'm just going to assume that this was a move and move its target counterpart accordingly instead of doing a delete+copy. " It's important to me that I be able to sync these machines quickly. Any suggestions?

I'd like to use Unison to sync a directory between my desktop computer and a container that lives in the cloud. I have this kind of working with Unison, but the problem I'm seeing is that every time I spin up the container, it can have a different hostname and IP number, and sshd will be running on a different port number. To make this mostly transparent to me, every time I spin up the container, I edit my ~/.ssh/config file like so:

Host pod
   User root
   Hostname 184.26.5.182
   Port 10666

I replace the IP number and port number accordingly. Once I've done this, I can log into the container using ssh pod, etc. I can run scripts that rsync stuff around with no problem, even though they have the host alias "pod" hardwired into them. (If I knew how to make rsync do a bidirectional sync with deletions, then I'd just use that, but I'm pretty sure that that is not possible.) My Unison config file looks like this:

root = /Users/me/path/to/dir
root = ssh://pod//path/to/dir
 
ignore = Path subdir1
ignore = Path subdir2

Unfortunately, this does not work great because every time I spin up the container, I get a different port number (the IP number doesn't usually change), and Unison considers this to be a fresh sync. Which is not at all what I want. Is there a way to tell Unison to ignore the fact that the remote container seems to have changed to a different computer, and to just trust that my "pod" host alias can be relied on for host identity?

I am syncing two machines using Unison: root = /home/luserface root = ssh://192.168.1.168/ I have a subdirectory "Foo" under that home, and it's a symbolic link. I have follow = Path Foo in my Unison profile, but that does not work; that Path doesn't reference Foo. What is the path I want here?

For over a decade now I have been working in a full Debian environment in my small office (currently 1 server, 7 users, 3 desktops, 4 laptops). Authentication is based on Kerberos, user profiles are managed in LDAP, and $HOME is served to all of the clients over NFSv4 with the help of pam_mount or autofs. This setup is very much ok for desktop users working on the local lan. Two years ago, I started using the same setup for laptop users. The WiFi connection caused some additional sluggishness, and for sure once the users tried to use the laptop outside of the office, things got really slow. Optimizing $XDG_{CACHE,DATA,CONFIG}_HOME and looking into specific optimizations for Firefox on NFS made things a bit better. I'm now thinking of moving the $HOMES to the laptops+desktops. It is nice that a user can switch devices if one goes down, but that only happens once in a while. Sacrificing this flexibility for a faster day-to-day user experience seems like a good decision. If I could bi-directionally sync the local $HOME to the central server on startup and shutdown, there probably wouldn't be a tradeoff at all... - 'unison' seems like a good candidate for keeping the local $HOME in sync with a central copy, but it seems to require exact same versions between server and client, and that I cannot commit to. - 'lsyncd' seems to be a very good candidate, but I don't seem to find any user stories using the tool for their $HOME directories... - I even had a brief look at 'GlusterFS', but that seems like it's a non-trivial replacement. Anyone has any experience and maybe best practices to share? I don't mind a bit of experimenting, but I'm afraid I'm missing some obvious downsides to the above... Thx!

unison-jz works well on a bash command-line but not inside a script. It seems that inside the script, it falls back to the regular unison command. Can someone tell me how to use unison-jz inside a bash script? ---------- I had the unison version conflict that is described in #583058 and the proposed solution for ubuntu works fine on an interactive shell. However, I want to do the same from inside a bash script: #!/bin/bash unison profile_name -sshargs '-i ~/.ssh/privatekey' This is a MWE version of the script.sh (in reality the script is more copmlex, but I checked this MWE and it shows the same faulty behaviour). When I invoke this script from the same command line as before (but also from another shell), it seems that inside the script, the regular unison version is used instead of the snapped unison-jz (which is used when invoking from the interactive shell). Hence, the syncing fails with the same *Fatal error* as before, saying "This can happen because you have different versions of Unison installed on the client and server machines", which I had fixed with the above approached. How can I make the shell script use the snapped unison-jz version instead of unison? Both client and server use bash shell on ubuntu (client: 20.04, server: 18.04). ---------- What I've done on the command-line: 1. Install snap: sudo snap install --devmode unison-jz 2. Add this line to my unison profile file: servercmd = /snap/bin/unison-jz.unison -ui text 3. Comment out line 19 in /etc/ssh/ssh_config so that is looks like # Include /etc/ssh/ssh_config.d/*.conf 4. Execute unison: unison profile_name -sshargs '-i ~/.ssh/privatekey' This works fine from an interactive bash shell.

I want to synchronize two computers that are similar and update just a few files. Therefore I would like to perform a Unison and ignore everything except a list of files in different paths. Something like: # /usr/bin/unison-gtk /home/ ssh://root@192.168.0.199:22//home/ -ignore "Name *" -ignorenot "Name {config/text.cfg, Pictures/test.jpg, Desktop/test.mp3" Or even better starting with root directory: # /usr/bin/unison-gtk / ssh://root@192.168.0.199:22// -ignore "Name *" -ignorenot "Name {/home/config/text.cfg, /home/Pictures/test.jpg, /usr/local/bin/test.sh" The first ignore is exclude everything else so the ignorenot won't catch them. Any Help? Can anyone help? Nothing I have found is helping me.

I originally [posted on SO](https://stackoverflow.com/questions/66462906/how-to-sync-specific-files-with-unison) but i realized this might be more suited here. I'm using [unison](https://www.cis.upenn.edu/~bcpierce/unison/) on Linux (Manjaro) to keep some files synced between two local folders but I'm running into a problem. I have a main notes folder which looks like this and has a lot of text files I write and export to PDF. my-notes/file1.pdf... file150.pdf I'd like to have some of these notes sent to another, more structured folder which I push to git.

organized-notes/
|
|------------------ Subject1/file34.pdf, file37.pdf
|
|------------------ Subject2/file11.pdf, file121.pdf
|
|------------------ Subject3/file97.pdf
|
|------------------ .git/

The issue is that unison works well between two directories or folders but I'm breaking down my files per subject and I would like them to be synced only in the relevant sub-folder. Also, I don't want to send all my notes to this more organized folder, only the specific ones I do select. Right now, anytime I make a change I have to cd to the relevant subject folder and use unison in there, like this unison ~/my-notes/file34.pdf file34.pdf. This is of course very annoying and I'd like to automate it. How I can fix this either with unison or another tool?

I have a small Debian ARM box with unison-2.40 installed. And a Ubuntu laptop which seems to have unison-2.48 I want to sync them with unison. But the the Ubuntu (which is newer) doesn't seem to recognise being explicitly asked to work with 2.40 (using -addversionno ) Is it possible to persuade 2.48 to pretend to be 2.40 or is there a Ubuntu package with 2.40 in it so I can sync these two machines?