Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once? I am particularly interested in calculating the MD-5 and SHA-256 digests of large files (size in gigabytes), preferably in parallel. I have tried openssl dgst -sha256 -md5, but it only calculates the hash using one algorithm. Pseudo-code for the expected behavior: for each block: for each algorithm: hash_state[algorithm].update(block) for each algorithm: print algorithm, hash_state[algorithm].final_hash()

I want to recursively delete all files that end with .in. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From this thread , it looks like it's possible to use xargs or make to parallelize find. Is this application of find possible to parallelize? Here is my current serial command: find . -name "*.in" -type f -delete

I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like puf < urls.txt does not work either. The operating system installed on the server is Ubuntu.

I have a script like this:

-shell
#!/bin/csh

command 1 \
   -f \"input1\" \
   -l input2 -other_swithes1

command 2 \
   -f \"input1\" \
   -m input2 \
   -l input3 -other_swithes1

command 3 \
   -f \"input1\" \
   -l input2 -other_swithes2

so any idea for an one-liner with xargs to run these commands in parallel. I tried various variants but all failed. I do not really want to write a script, I think that should have been possible with -d switch and -c, not sure though. To simplify and extend the problem further, what I have is

cat file | grep -v "#.*" | sed -z 's/[\]\n/ /g' | xargs -I {} -n1 -P10 sh -c '{}'

and while this does the job there is a particular problem and that is that \" get removed. So any clue how to solve that?!

I am brand new to zstd/pzstd, trying out its features, compression, benchmarking it, and so on. (I run Linux Mint 22 Cinnamon.) This computer has 32 GB RAM. The basic command appears to be working, but I found out it's not fully multi-threaded / parallelized:

# zstd --ultra --adapt=min=5,max=22 --long --auto-threads=logical --progress --keep --force --verbose /dev/nvme0n1 -o /path/to/disk/image/file.zst

As you can see for yourself, I am trying to compress my **NVMe 4TB drive** with only Timeshifts on its ext4 fs. Could you recommend me some tweaks to my zstd command, I would welcome it. But the real question here is how to make it multi-threaded / parallelized? *** I am trying this:

# pzstd --ultra -22 --processes 8 --keep --force --verbose /dev/nvme0n1 -o /path/to/disk/image/file.zst

because of this parallel version of ZSTD does not have apparently the --progress option. I need to find another way to watch it. 4TB will take some time and I don't intend to be totally blind. My tries with pv ended as not working properly. Please help, I'll appreciate it. Thanks.

I've a problem modifying the files' names in my Music/ directory. I have a list of names like these: $ ls 01 American Idiot.mp3 01 Articolo 31 - Domani Smetto.mp3 01 Bohemian rapsody.mp3 01 Eye of the Tiger.mp3 04 Halo.mp3 04 Indietro.mp3 04 You Can't Hurry Love.mp3 05 Beautiful girls.mp3 16 Apologize.mp3 16 Christmas Is All Around.mp3 Adam's song.mp3 A far l'amore comincia tu.mp3 All By My Self.MP3 Always.mp3 Angel.mp3 And similar and I would like to cut all the numbers in front of the filenames (not the 3 in the extension). I've tried first to grep only the files with the number with find -exec or xargs but even at this first step I had no success. After being able to grep I'd like doing the actual name change. This is what I tried by now: ls > try-expression grep -E '^[0-9]+' try-expression and with the above I got the right result. Then I tried the next step: ls | xargs -0 grep -E '^[0-9]+' ls | xargs -d '\n' grep -E '^[0-9]+' find . -name '[0-9]+' -exec grep -E '^[0-9]+' {} \; ls | parallel bash -c "grep -E '^[0-9]+'" - {} And similar but I got error like 'File name too long' or no output at all. I guess the problem is the way I'm using xargs or find as expressions in separate commands work well. Thank you for your help

**Question:** Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with multiple requests at different file offsets? This could help with performance of each socket is throttled separately. I could write a program to do this, but I'm wondering if the tooling already exists. **Background:** Recently I wanted to download a large iso, **but!** ... Somewhere between the server and my internet provider the transfer rate was limited to 100 kilobit! However, I noticed that the first 5 to 10 seconds had great throughput, hundreds of megabits. So I wrote a small bash script to restart after a few seconds: while ! timeout 8 wget -c http://example.com/bigfile.iso ; do true; done (I hope it was not my provider . . . But maybe it was. Someone please bring back net neutrality!)

I have this script to go through a list of URLs and the check return codes using Curl. Links file goes like this: https://link1/ ... https://link2/ ... https://link200/ ... (...) The script: INDEX=0 DIR="$(grep [WorkingDir file] | cut -d \" -f 2)" WORKDIR="${DIR}/base" ARQLINK="navbase.txt" for URL in $(cat $WORKDIR/$ARQLINK); do INDEX=$((INDEX + 1)) HTTP_CODE=$(curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' $URL) if [ $HTTP_CODE -eq 200 ]; then printf "\n%.3d => OK! - $URL" $INDEX; else printf "\n\n%.3d => FAIL! - $URL\n" $INDEX; fi done It takes a little while to run through every URL, so I was wondering how to speed those up curl requests. Maybe I could use some parallel Curl requests, but using "xargs" inside a "for" loop while also printing a message doesn't seem the way to go. I was able to use "xargs" out of the script and it sort of works, although not showing the correct HTTP code. cat navbase.txt | xargs -I % -P 10 curl -m 5 -k -o /dev/null --silent --head --write-out '%{http_code}\n' % I couldn't find a way to insert that into the script. Any tips?

I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script? Is the output of nproc equivalent to the number of threads I have available to compile with? make -j1 make -j16

In a slurm system, when I use **srun** command to run the program. It runs very slow and seems like only one processor works. srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16 $FEAPHOME8_3/parfeap/feap -log_summary lu.log But if I write a **sbatch** script, it can run very quickly and looks like all the processors work. #!/bin/sh -l #SBATCH --job-name=test #SBATCH --account=free #SBATCH --nodes=1 #SBATCH --ntasks-per-node=24 #SBATCH --cpus-per-task=1 #SBATCH --exclusive #SBATCH --time=6:00:00 echo ' ' echo ' ****** START OF MAIN-JOB ******' date srun -n 16 echo y | mpirun -np 16 $FEAPHOME8_3/parfeap/feap -log_summary lu.log echo ' ****** END OF MAIN-JOB ******' #End of script Could anybody please tell me what's going on?

Using Bash in OSX. My script has these 2 lines: nfiles=$(rsync -auvh --stats --delete --progress --log-file="$SourceRoot/""CopyLog1.txt" "$SourceTx" "$Dest1Tx" | tee /dev/stderr | awk '/files transferred/{print $NF}') & nfiles2=$(rsync -auvh --stats --delete --progress --log-file="$SourceRoot/""CopyLog2.txt" "$SourceTx" "$Dest2Tx" | tee /dev/stderr | awk '/files transferred/{print $NF}') When I use the & after the first line (to run the two rsync commands in parallel), my later call to $nfiles returns nothing. Code: osascript -e 'display notification "'$nfiles' files transferred to MASTER," & return & "'$nfiles2' transferred to BACKUP," & return & "Log Files Created" with title "Copy Complete"' Can't figure out what's going on. I need the 2 rsyncs to run simultaneously.

I am computing Monte-Carlo simulations using GNU Octave 4.0.0 on my 4-core PC. The simulation takes almost 4 hours to compute the script for 50,000 times (specific to my problem), which is a lot of time spent for computation. I was wondering if there is a way to run Octave on multiple cores simultaneously to reduce the time of computations. Thanks in advance.

I'm a bit confused about concurrency and parallelism in the bash shell. As I understand it, when we run commands in more than one subshells at the same time, these commands run in parallel on individual processor cores. **For example;**

cmd1 & cmd2 & cmd3 &

Here, the "ampersand" sign is run in the background (aka the subshells) of each command at the same time.It could be create in subshells in other ways. (Like writing in parentheses or using a pipe ..). In this direction, the questions I wonder the answers of; - Bash provides parallelism through subshells, other hand is concurrency also achieved using another method on bash? As far as I know, concurrency works in a way that a single CPU executes operations intermittently. Do I need to implement a method externally to achieve this, or is bash already working this way (concurrency ) anyway. - If I occupy all CPU cores using parallelism, will the system crash or is there a protection mechanism against this situation? - What is the difference between the parallel I provide with subshells and the GNU Parallel tool? If the GNU Parallel tool works better, how does it achieve this? - Which of the "Parallel" or "Concurrency" operations works more efficiently? - What kind of losses are experienced when making "parallel" or "concurrency" operation, unlike normal (sequential execution of commands)?

Normally, pipelines in Unix are used to connect two commands and use the output of the first command as the input of the second command. However, I recently come up with the idea (which may not be new, but I didn't find much Googling) of using pipeline to run several commands in parallel, like this: command1 | command2 This will invoke command1 and command2 in parallel **even if command2 does not read from standard input and command1 does not write to standard output**. A minimal example to illustrate this is (please run it in an interactive shell) ls . -R 1>&2|ls . -R My question is, are there any downsides to use pipeline to parallelize the execution of two commands in this way? Are there anything that I have missed in this idea? Thank you very much in advance.

I have 2 commands which are to be run simultaneously. And I want the script to terminate if one of them either exits with code 0 or 1. How can I achieve this in Linux(Ubuntu) cmd1 & cmd2 & wait

I have a command which I want to run on all free cores to speed up the execution time. Specifically I am running the Pitman-Yor Adaptor-Grammar Sampler software I downloaded from here ./py-cfg/py-cfg-mp -r 0 -d 10 -x 10 -D -E -e 1 -f 1 -g 10 -h 0.1 -w 1 -T 1 -m 0 -n 500 -G x.tgt y.tgt < z.tgt I tried adding parallel -j "$(nproc)" before the command as specified in this answer but it is generating the following error: Error in ./py-cfg/py-cfg-mp, argc = 29, optind = 27

I created some pigz (parallel gzip) - home page - compressed archives of my SSD disk drives. (compiled version 2.8) I called one of them **4TB-SATA-disk--Windows10--2024-Jan-21.img.gz** which says the size of the drive and the OS installed on it. Alternatively, the size is of course smaller in TiB which may be comfortably shown by fdisk and tools like it (Disk /dev/sda: 3.64 TiB). I knew from the past, that listing the compressed file cannot show me the real size of contents. It will show 2.2GB or similar nonsense, even in Archive Manager for GNOME. It likely has something to do with gzip structure limitations. However I had some doubts about the real size of the contents, therefore my question states how may I verify it?

I have been trying to understand lscpu's output and came across several threads dedicated to concepts of CPUs, physical cores, and threads. Based on those threads, to get the total number of CPUs (logical units), you would do the following: Thread(s) per core x Core(s) per socket x Socket(s). Given the partial output from my machine shown below, I would expect CPU(s) to be 28, but it is listed as 20.

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  20
  On-line CPU(s) list:   0-19
Vendor ID:               GenuineIntel
  Model name:            12th Gen Intel(R) Core(TM) i9-12900H
    CPU family:          6
    Model:               154
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):           1

Can someone help me to understand the apparent disconnect between my expectation for the CPU count and what lscpu actually produced in this instance? I can't seem to find similar cases.

So at work we have a few systems running on their own hardware that is identical to all of the systems. If one of the hardware fails, one of the running system will take over and start the system that failed, doing the work of both. All Linux and Intel cpu based. How would that be possible to have a OS start another OS to run together? It is not just running a program, it is running the entire thing as well. I have heard of things like how you can have windows let another OS use one of the cores on the cpu for parallel computing. But I am curious to how a system would immediately take over another system. It was super cool and my co workers unfortunately don't understand how it is done either.

I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel My current code is working fine for limited entries in the reg2 variable. But I have millions of entries in the reg2 variable. So I want to make my outermost loop parallel. To get the same output i.e., 0,1,2,:,3,4,:,5,6, after parallelizing the code #!/bin/bash # array1=$1 # array2=($2) # reg2=($3) array1=('bam1' 'bam2' 'bam3' 'bam4' 'bam5' 'bam6' 'bam7') array2=('cell1' 'cell1' 'cell1' 'cell2' 'cell2' 'cell3' 'cell3') reg2=('chr1:10484-10572' 'chr1:10589-10632' 'chr1:10636-10661' 'chr1:10665-10690' 'chr1:10694-10719') start=date +%s.%N l=${#reg2[@]} # number of regions is 30 million on real data reg_cov=() j=0 for r in ${reg2[@]}; do (cov_array=() old_array2_element=${array2} for i in ${!array1[*]}; do new_array2_element=${array2[$i]} if [[ "$new_array2_element" != "$old_array2_element" ]]; then cov_array+=(":") old_array2_element=$new_array2_element fi cov_array+=($i) # in actual code this step takes 4-5 seconds to process sleep 2 done reg_cov+=($(IFS=, ; echo "${cov_array[*]}")) ) wait ((j++)) echo "$j/$l" done #echo ${reg_cov[@]} cov=() cov+=(${reg_cov[@]}) echo $cov end=date +%s.%N; runtime=$( echo "$end - $start" | bc -l ); runtime=${runtime%.*}; hours=$((runtime / 3600)); minutes=$(( (runtime % 3600) / 60 )); seconds=$(( (runtime % 3600) % 60 )) echo "==> completed Runtime: $hours:$minutes:$seconds (hh:mm:ss)"