So I am trying to calculate a SHA1 digest (and base64 encode it) for an attachment I received so it matches their (base64 encoded) digest. Using ChatGPT, I was able to use the command below to succeed. But I don't understand the command xxd -r -p or why it's necessary. sha1sum foo | awk '{print $1}' | xxd -r -p | base64 There's two things I don't understand about it: 1) Why is the option -p needed? I understand xxd -r makes a binary dump of a hexdump, but if I leave off the -p it prints nothing while the man page says it should print to stdout and that -p is just for a particular output style. 2) What is the purpose of even passing the SHA1 digest through xxd? My understanding is xxd essentially converts data between hex and binary. But when I run the above command without xxd -r -p, it gives a different, and unmatched, output. I do know what hex style/representation is (that's the format of a SHA1 digest), but perhaps I don't know what "binary" means in this context?

Is it possible to have SHA sums print without the - appended to the end? $ echo test | sha1sum 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 - <--- this "-" dash/hyphen I know we can use awk and other command line tools, but **can it be done without using another tool**? $ echo test | sha1sum | awk '{print $1}' 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83

I have a series of directories with sha1sums and md5sums files in. The format of these files is the usual hash space space filename, with one hash/file per line. I want to verify the files and print out the path as well as the filename of corrupt files.

find . -name SHA1SUMS -execdir echo "$PWD" sha1sum --quiet --check SHA1SUMS \; > logfile

(modified from here ) gives

./path1/SHA1SUMS
sda2.ntfs-ptcl-img.gz.aa: FAILED
blkdev.list: FAILED
Info-dmi.txt: FAILED
./path2/SHA1SUMS

Whereas I am looking more for a

./path1/sda2.ntfs-ptcl-img.gz.aa: FAILED
./path1/blkdev.list: FAILED
./path2/file: FAILED

type of output.

Perhaps I'm doing something very stupid here, but Google hasn't been very helpful in resolving the problem. I have an archive of files I made for backup purposes. I've generated an SHA1 checksum file from this archive: sha1sum myarchive.tar > myarchive.tar.sha1 The contents of this file are as follows: 6f5d7bdd71fe25ed8e881265fdb8a8bbcdaa41c1 myarchive.tar I have also run the SHA1 process in the terminal without piping to a file: sha1sum myarchive.tar This gives me the output: 6f5d7bdd71fe25ed8e881265fdb8a8bbcdaa41c1 myarchive.tar Clearly these checksums are the same. However, when I run the verification command, with the archive and its SHA1 file next to each other in the same directory: sha1sum -c myarchive.tar.sha1 I get an error message saying that the checksum does not match: myarchive.tar: FAILED sha1sum: WARNING: 1 computed checksum did NOT match Obviously something is wrong here, but I have no idea what it could be. Can anyone enlighten me? EDIT: Interestingly, doing two consecutive MD5s on the file comes up with two different checksums. Now I'm very confused. $ md5sum myarchive.tar 9a15036eed341613bbcf2c4b53a09859 myarchive.tar $ md5sum myarchive.tar a662d6b469627c62f2b03ee0df067436 myarchive.tar EDIT2: Additional context: * This is on real hardware (my Ubuntu MATE 19.10 desktop machine). * The archive I've made was destined for a Blu Ray backup disc. It's 22.6GB in size. * Performing an SHA1 verification of the file as burned to the Blu Ray disc ends up being successful. EDIT3: In response to requests to see the output of dmesg, there appear to be a few errors like the following: [ 7102.039819] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 [ 8278.017874] sr 4:0:0:0: [sr0] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 8278.017876] sr 4:0:0:0: [sr0] tag#0 Sense Key : Blank Check [current] [ 8278.017877] sr 4:0:0:0: [sr0] tag#0 Add. Sense: No additional sense information [ 8278.017878] sr 4:0:0:0: [sr0] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00 [ 8278.017879] blk_update_request: critical target error, dev sr0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 8278.019391] sr 4:0:0:0: [sr0] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 8278.019392] sr 4:0:0:0: [sr0] tag#0 Sense Key : Blank Check [current] [ 8278.019392] sr 4:0:0:0: [sr0] tag#0 Add. Sense: No additional sense information [ 8278.019393] sr 4:0:0:0: [sr0] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00 [ 8278.019394] blk_update_request: critical target error, dev sr0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 8278.019396] Buffer I/O error on dev sr0, logical block 0, async page read I think this is to do with my USB Blu Ray drive, though someone may be able to correct me.

I tried with openssl as it is very fast:

openssl sha1 "$(basename "very_big_image.png")"

But this just substitutes the actual file for check summing.

You can argue that you can check the checksum of the application's package, but you are then trusting the developer to have compiled the right code and provided the right signature. Is there a way to verify this trustlessly, or is it impossible? Thanks!

I read that using the sha1sum command will run the SHA-1 algorithm and give you a "unique" result, but how can that be? sha1sum gives you a 40 character hash, like this: e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e This is 40 characters long, and has hexadecimal characters, so the hash can have at most 1640 combinations. This should mean that, for any two random files, there *is* a small chance that their hash sum would be exactly the same, right?