Is it possible to compare two directories with rsync and only print the differences? There's a dry-run option, but when I increase verbosity to a certain level, every file compared is shown. ls -alR and diff is no option here, since there are hardlinks in the source making every line different. (Of course, I could delete this column with perl.)

OK, I will try and explain what I need to do as best as possible. Basically I have two CSV files, as per the examples below: File 1: Column 1, Column 2 abc , 123 def , 234 adf , 567 File 2 Column 1, Column 2 abc , 123 def , 234 adf , 578 I need to write either a shell script or simple command that will do the following: 1. Sort both files by column 1 2. Row by row, do the following: - Using column 1 in file 1, search for this value in column 1 in file 2. 1. if found, compare the value in column 2 in file 1 against the value in column 2 of file 2 1. if it matches, write column 1, column 2 and "Validated" in column 3 to a separate file 1. if it does not match, write column 1, column 2 and "Failed" to a separate file This results in two output files: the first with everything that was found in column 1 and column 2 matches, and a second file containing either column 1 lookups that failed or where column 1 was found, where column 2 did not match, so, essentially, using column 1 as the key to check column 2.

I have two files file1.csv and file2.csv. Below are the contents of file1.csv AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AR.jar;9a553dd203d0979aa60004e19cc98c12 BI.jar;8022f6c5f83ba040394ff0b0a0323e8e BV.jar;f53c4a8c988aa8806b54063ebc682803 CaseUtilities.jar;e5f653d899298f5e5d56f357b6f781c5 CO.jar;b2f7a0ab6e646d6793631e5c97e05096 file2.csv AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AR.jar;4e6e584dd852684ba21ae63990e2a1a6 BV.jar;213d9df82095764702ef4929424a1a0c CaseUtilities.jar;5b787f1f3d57922bd980ebbfe9a5343e CO.jar;cfb994078ff4373c7e0f15de19830a3d Common.jar;a09b520288870aa3888194ce59179dbd We need to compare two files based on the contents. I want to make diff which is based only on values of first column, so the result should be AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AR.jar;9a553dd203d0979aa60004e19cc98c12 AR.jar;4e6e584dd852684ba21ae63990e2a1a6 BI.jar;8022f6c5f83ba040394ff0b0a0323e8e BV.jar;f53c4a8c988aa8806b54063ebc682803 BV.jar;213d9df82095764702ef4929424a1a0c CaseUtilities.jar;e5f653d899298f5e5d56f357b6f781c5 CaseUtilities.jar;5b787f1f3d57922bd980ebbfe9a5343e CO.jar;b2f7a0ab6e646d6793631e5c97e05096 CO.jar;cfb994078ff4373c7e0f15de19830a3d Common.jar;a09b520288870aa3888194ce59179dbd I have tried the below command diff -y file1.csv file2.csv But the below output is not as expected. AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AL.jar;d8c06ebedd7954681f34ab5c94fdc4fb AR.jar;9a553dd203d0979aa60004e19cc98c12 | AR.jar;4e6e584dd852684ba21ae63990e2a1a6 BI.jar;8022f6c5f83ba040394ff0b0a0323e8e | BV.jar;213d9df82095764702ef4929424a1a0c BV.jar;f53c4a8c988aa8806b54063ebc682803 | CaseUtilities.jar;5b787f1f3d57922bd980ebbfe9a5343e CaseUtilities.jar;e5f653d899298f5e5d56f357b6f781c5 | CO.jar;cfb994078ff4373c7e0f15de19830a3d CO.jar;b2f7a0ab6e646d6793631e5c97e05096 | Common.jar;a09b520288870aa3888194ce59179dbd Any idea how can my expected output be achieved!

With

diff -r

I can do this task, however it takes so long because diff checks file's content. I want something that determine that two files are the same regarding of their size, last modified, etc. But no checking bit by bit the file (for example a video takes sooo long). Is there any other way?

In a Unix command line context I would like to compare two truly huge files (around 1TB each), preferable with a progress indicator. I have tried diff and cmp, and they both crashed the system (macOS Mojave), let alone giving me a progress bar. What's the best way to compare these very large files? ### Additional Details: 1. I just want to check that they are identical. 2. cmp crashed the system in a way that the system did restart by itself. :-( Maybe the system ran out of memory?

I want to be able to compare 2 excel sheets in linux. I am not interested in converting them to csv format as they have a complicated formating that is not supported in csv. I would like to be able to have a graphical comparison (some sort of kompare way). The tool should be available in linux platform. Any ideas ?

I would like to check when a file differ inside a backup system using snapshots. I have several folders with the same architecture inside ls -1 .snapshot 4-hourly.2024-04-14_0405 4-hourly.2024-04-14_0805 4-hourly.2024-04-14_1205 4-hourly.2024-04-14_1605 4-hourly.2024-04-14_2005 4-hourly.2024-04-15_0405 4-hourly.2024-04-15_0805 4-hourly.2024-04-15_1205 daily.2024-04-08_0010 daily.2024-04-09_0010 daily.2024-04-10_0010 daily.2024-04-11_0010 daily.2024-04-12_0010 daily.2024-04-13_0010 daily.2024-04-14_0010 daily.2024-04-15_0010 monthly.2024-01-01_0020 monthly.2024-02-01_0020 monthly.2024-03-01_0020 monthly.2024-04-01_0020 weekly.2024-02-25_0015 weekly.2024-03-03_0015 weekly.2024-03-10_0015 weekly.2024-03-17_0015 weekly.2024-03-24_0015 weekly.2024-03-31_0015 weekly.2024-04-07_0015 weekly.2024-04-14_0015 And I have to check a file located inside each of these folder. For example, the goal is to see if .snapshot/weekly.2024-04-14_0015/my/path/to/the/file.php differs from weekly.2024-04-07_0015/my/path/to/the/file.php or from .snapshot/weekly.2024-03-31_0015/my/path/to/the/file.php, or from .snapshot/weekly.2024-04-07_0015/my/path/to/the/file.php etc. Is there an obvious simple way for this? P-S : There are other files/folders that changed inside this folder and I cannot just compare the whole folder.

I am doing diff -rq to check the difference between 2 folders (A and B). I don't want to know if folder A has any new files because that's expected. How do I ensure that this information doesn't come up?

Referring to this https://stackoverflow.com/a/31356602 , I wrote this code:

#!/bin/bash

# Define the two strings to compare
string1="First string with some random text."
string2="Second string with some random text and some changes."

# Create a temporary directory
temp_dir=$(mktemp -d)

# Create temporary files for the strings
file1="$temp_dir/string1.txt"
file2="$temp_dir/string2.txt"
echo -e "$string1" > "$file1"
echo -e "$string2" > "$file2"

# Use the git diff command to compare the temporary files
git diff --no-index --word-diff=color --word-diff-regex=. "$file1" "$file2"

# Delete the temporary directory
rm -rf "$temp_dir"

that returns: Now I'm trying to condensate it in a single line:

#!/bin/bash

# Define the two strings to compare
string1="First string with some random text."
string2="Second string with some random text and some changes."

# Use the git diff command to compare the strings
git diff --no-index --word-diff=color --word-diff-regex=. <('%s\n' "$string1") <(printf '%s\n' "$string2")

but I get: How can I pass strings as files to git diff without explicitly creating temporary files? **Note.** My goal is to "visually" compare (character-level) two (short) strings, obtaining an output similar to this: in which the differences between the two compared strings are highlighted in a single string. The output of git diff is ideal, but I am also open to other solutions.

Is there a way/tool to show differences between two files side by side only highlighting differences in a similar way kdiff3 does that but with a tool that is reasonably fast?A similar question was asked here almost 6 years ago but no useful answers given. Even better a tool that understands data structure such as csv? I often deal with data files that might differ in a single column by a sign only or format of the number and for this purpose diff is quite useless. kdiff3 shows the data like this: Which immediately tells you what is different between the data files. It not only shows you that the whole line is different but also what is different. But, kdiff3 is insanely inefficient, comparing 2 17 MB files takes 10 minutes which is beyond ridiculous. (I once tried something called total commander on windows which has this function and its comparing efficiency blew my mind). Being aware of data structure and showing only the different fields would be a huge advantage but all csv comparators that I found cannot deal with a situation that one of the files has different number of columns (say you add a column to your data) and they can only compare the same structure and so i am good with the kdiff3 or total commander style approach. I tried vimdiff --- says it cannot run diff on the files, meld --- slow and in the end does nothing, icdiff -- slow, etc. Just as comparison and an illustration what i am looking for, i reached to a virtual machine compared the same two files with total commander (over network connection) and in 2 seconds (compared to 10 minutes of kdiff3 on local ssd), i got:

I have two lists with item numbers and want to mark the difference between these lists by writing the numbers wich aren't in both files in a new file. Both Files have the item number in column 2 and part ID in column 3. First I want to check if the item number in file 1 exists , if so the part ID has to be checked and if it's true, go to the next item number. Otherwise, if one of the conditions isn't true the difference should be written in a new created file. In case that a item number is only in one of the two files the program should write "Artikel[x] cannot be found". EXAMPLE FILE 1 Artikel[ 456]= 1,2 Artikel[ 877]= 3 Artikel[ 278]= 4 Artikel[ 453]= 13 FILE 2 Artikel[ 456]= 2, 1 Artikel[ 877]= 3, 5 Artikel[ 387]= 4, 9, 4 Artikel[ 947]= 10 OUTPUT Artikel[ 877]= 3 != Artikel[ 877]= C3, C5 Artikel[ 278]= 4 != Artikel[ 278 ]= 4, 9, 4 Artikel[ 453]= 13 cannot be found in File 2! Artikel[ 947]= 10 cannot be found in File 1! I thought i could do this with writing the item number from File 1 in an array and just check every line in File 2 but somehow I am struggling to manage that. I would appreciate any help. Thanks

I have two directories with files with the same name. Basically I want to replace dir1/file.txt with dir2/file.txt if the creation date of dir1/file.txt is more recent. But i think I'm missing something in the if condition. #!/bin/bash for i in /dir1/*; do nameFIRR=$(basename "$i") dateINPUT="$(date -r $i)" dateOUTPUT="$(date -r /dir2/$nameFIRR)" if [ $dateINPUT -ge $dateOUTPUT ]; then cp -u $i /dir2/$nameFIRR fi done

I am trying to find a way to determine if a text file is a subset of another.. For example: foo bar is a subset of foo bar pluto While: foo pluto and foo bar are not a subset of each other... Is there a way to do this with a command? This check must be a cross check, and it has to return: file1 subset of file2 : True file2 subset of file1 : True otherwise : False

I have a lot of audio files to convert and I like this to happen automatically when moving a folder with files into another folder, where the magic should happen. It's on a debian system. Any help would be greatly appreciated.

I'm currently encountering some challenges while working on a bash script in the Linux terminal to perform the following tasks: 1. Compare the values in the third column of two different files line by line. 2. If the values are different, save in a third output file the identifier from the second column of the input files, along with the corresponding values from both File 1 and File 2, similar to the "desired first output" example provided. 3. Additionally, create a fourth output file to tally the occurrences of each unique qualitative difference, taking into account the order (for example: 9690 0 ≠ 0 9690) while disregarding the identifier. This is illustrated in the "desired last output" example. Any assistance or guidance in achieving this is greatly appreciated! FILE 1

U	E100033877L1C016R01601996031		0	140	0:106
U	E100033877L1C023R03303214633		0	140	0:106
C	E100033877L1C022R01901579971		27996	140	27996:1 0:7 27996:23 0:75
C	E100033877L1C023R02603225407		27996	140	0:32 27996:23 0:7 27996:1 0:3 27996:4 0:36
C	E100033877L1C020R02602000209		0	140	0:106
C	E100033877L1C023R03303214633		27996	140	27996:3 0:4 27996:5 0:94
C	E100033877L1C023R03101740491		9690	140	9690:13 0:8 9690:7 0:13 9690:9 0:56
C	E100033877L1C006R00200498634		9690	140	9690:71 0:35
C	E100033877L1C009R03603066069		27996	140	0:50 27996:2 0:1 27996:10 0:6 27996:11 0:26
C	E100033877L1C005R03300436825		27996	140	27996:3 0:6 27996:3 0:3 27996:5 0:86

FILE2

U	E100033877L1C016R01601996031		0	140	0:106
U	E100033877L1C023R03303214633		0	140	0:106
C	E100033877L1C022R01901579971		27996	140	27996:1 0:7 27996:23 0:75
C	E100033877L1C023R02603225407		27996	140	0:32 27996:23 0:7 27996:1 0:3 27996:4 0:36
C	E100033877L1C020R02602000209		27996	140	0:19 27996:4 0:3 27996:1 0:7 27996:23 0:49
C	E100033877L1C023R03303214633		27996	140	27996:3 0:4 27996:5 0:94
U	E100033877L1C023R03101740491		0	140	0:106
U	E100033877L1C006R00200498634		0	140	4840:106
C	E100033877L1C009R03603066069		4840	140	0:50 27996:2 0:1 27996:10 0:6 27996:11 0:26
C	E100033877L1C005R03300436825		27996	140	27996:3 0:6 27996:3 0:3 27996:5 0:86

DESIRED FIRST OUTPUT

E100033877L1C020R02602000209		0	27996
E100033877L1C023R03101740491		9690	0	
E100033877L1C006R00200498634		9690	0	
E100033877L1C009R03603066069		27996	4840

DISIRED LAST OUTPUT

2	9690	0	
1	0	27996
1	27996	4840

I'd like to compare directories with binary files. Actually, I'm not interested in what the actual differences between files are, but to know if there's a differ (and what files differ). Previously I used meld, but it's cannot compare binary files. What such file comparison tool can do this? **NOTE:** It doesn't matter if it's a graphical tool or is just has a command-line.

[ EDIT: Answered my own question. Used unison and some hacky post-processing ( copy-pasted the log output of unison, tweaked it in my text-editor with multi-selection editing, then did some shell script processing on that (fishshell) ) ((I've got so much more disaster recovery stuff to slog through, so I guess I'm done with *this* for now...)) ] --- So I have an SSD that used to be inside another computer, and I put it in one of those little SATA-to-USB enclosure/adapter shell things, mounted it as an external data drive, and used rsync -aAX to copy the boot partition over into a dir on this computer, for a backup. But then, after some other events that *probably* didn't change the contents of the original boot partition, I made a second backup. So now I have two dirs on this computer, which I think are *probably* two copies of exactly the same backup, but I want to make sure. --- So my question is: **What is the best way to compare/diff these two big backup directories?** Summary of things I have considered/tried but had problems with or am unsure about: - diff itself - rsync "dry run" trick - unison [(just thought of, but it hasn't finished running yet, due to the large size of the backups and the slow speed of my old hardware.)] Is one of these essentially a good option? If so, any corrections to the details of how I should go about using it? Or are there any separate, additional options I should know about..? --- # Details of my attempts and the results/problems: --- ## diff The obvious way of doing it for "*normal*" directories would be like: $ diff -r dir_A dir_B ( or maybe $ diff -r --no-dereference dir_A dir_B? I don't know; I honestly don't properly understand the function of --no-dereference -- it's just something I've found got me the results I wanted in *vaguely* similar situations in the past. ) However, the problem with using diff is that these dirs are of course very unusually large, and full of "weird" files from the bootable system (eg "character special files" and "block special file" etc). --- ## rsync So it occurred to me to just use rsync again between them, doing a "dry run" and seeing if it reported any changes it would make. Like: $ sudo rsyncy -n -aAX dir_A dir_B --log-file=log_file However, it then occurred to me - "what if there were *new* files in dir_B?" - "would rsyncy necessarily always report that?" So I guessed you would have to check both: $ sudo rsyncy -n -aAX --delete dir_A dir_B --log-file='log_file[A-to-B]' and $ sudo rsyncy -n -aAX --delete dir_B dir_A --log-file='log_file[B-to-A]' which is starting to feel a little suspiciously like maybe that isn't really the right tool for the job after all...? The log files I got read: A-to-B #=>

2023/07/21 01:43:04  building file list  
2023/07/21 02:12:24  sent 80.58M bytes  received 292.46K bytes  45.93K bytes/sec  
2023/07/21 02:12:24  total size is 229.29G  speedup is 2,835.29 (DRY RUN)

B-to-A

2023/07/21 01:41:58  building file list  
2023/07/21 02:12:15  sent 80.58M bytes  received 292.50K bytes  44.49K bytes/sec  
2023/07/21 02:12:15  total size is 229.29G  speedup is 2,835.29 (DRY RUN)

which are (ignoring the timestamps and speed information), annoyingly *ALMOST* the exact same: both sent 80.58M bytes but *tiny* different received: received 292.46K bytes vs received 292.50K bytes So yeah, again, I'm feeling doubtful this rsync trick is really the right tool for the job...? Maybe the correct answer really *is* like: " Just be patient and let diff run for ages to process the two huge directories. (You can just ignore all the error messages about special file etc.) " ?? --- ## unison [not sure yet?]

**Background**: I use Obsidian to organize my notes as I study. It's an application which works "on top of" a collection of markdown formatted plain text files and then shows the links and connections between them graphically. The text files I stored within sub folders of a single folder **What I want:** Since I am studying for an upcoming exam, I add information to several of these text files each day. At the end of the day, I'd like to have an automatically generated single text file containing all the changes I made to each of my files. A kind of "daily digest" which I can then read through to revise everything I learned on that day. **Additional background:** I already have a cron job set up to use rsync to make an uncompressed copy of my notes folder every day at 3:00 AM. I assume this copy can act as a reference when I want to see what modifications have been made at the end of the current day. **An example** Let's say on the 15th of May, I made the following changes Added "This is some new text" to file A Added - "this is some more textual information" to file B. My "daily digest" at the end of the 15th of May would read "This is some new text" This is some more textual information" **My Research:** I understand that tools like diff and meld (GUI) are great for comparing files. The diff output formatting it a quite difficult to understand. Meld on the other hand is very easy to use but it's a GUI app so I can't figure out how to direct it's output to a single text file.

I have 2 directories $ tree dir{1..2} dir1 ├── file1 └── file2 dir2 ├── file1 └── file2 I want to compare all files in dir1 with all files in dir2 using ansible and print differences like this output: ${file1} from ${dir1}: diff content ${file1} from ${dir2}: diff content and it will loop through all files to print their differences below is ansible snippet that needs modification --- - name: Compare files in two directories hosts: localhost gather_facts: false tasks: - name: Find files in directory 1 find: paths: ~/dir1 file_type: file register: dir1_files - name: Find files in directory 2 find: paths: ~/dir2 file_type: file register: dir2_files - name: Compare files shell: diff dir1/file1 dir2/file1 ## how can I make sure path changes but filenames stay same using variables loop: "{{ dir1_files.files }}" register: diff_output changed_when: false failed_when: diff_output.rc == 1 - name: Print differences debug: msg: | {{ item.item.path }} from dir1: {{ item.stdout }} {{ item.item.path }} from dir2: {{ item.stdout }} loop: "{{ diff_output.results }}" when: item.stdout_lines | length > 0 For suggested code in Vladimir's answer, I get below output TASK [debug] ***************************************************************************************************************************************** ok: [localhost] => { "msg": "file2 from dir1: \n 1,2c1,2\n abc101\n > def111\nfile2 from dir2: \n 1,2c1,2\n abc101\n > def111\nfile1 from dir1: \n 1,2c1,2\n 101abc\n > 111def\nfile1 from dir2: \n 1,2c1,2\n 101abc\n > 111def\n" }

I have the code below (please see https://unix.stackexchange.com/questions/392722/need-to-understand-below-awk-command-to-find-missing-lines-in-a-file/744316?noredirect=1#comment1415306_744316) : awk 'NR==FNR{a[$0];next}(!($0 in a)){print}' 1.txt 2.txt Can I add one more condition to skip the comparison of lines if the line starts with = (i.e '$0 ~ /^=/ {print $0}') and just print those lines alone as it is?