We are in a concurrent scenario where we have **n** concurrent processes. By using a synchronization policy (e.g., using pipes or signals), each process can print using *printf("string\n")* only if it receives a token from the previous process, ensuring a specific printing order. When stdout is attached to a terminal (e.g., a console), the expected printing order is respected, meaning processes print in the exact order defined by the synchronization policy. However, when stdout is piped or redirected to a file or another process, the resulting printing order may not be respected, even though the processes still follow the synchronization policy correctly. We know that *printf* is buffered by default, meaning it does not immediately write to stdout but instead accumulates output in a buffer before flushing it. In a concurrent environment, it seems the operating system writes each process's buffer in an order independent of the intended synchronization policy, causing out-of-order prints. This is unexpected given that each process follows the policy correctly. However, this problem does not occur if one of the following solutions is implemented: 1. Forcing a flush after each call to printf using *fflush(stdout)*; 2. Disabling buffering for stdout in each child process using setvbuf(stdout, NULL, _IONBF, 0);. 3. Using the system call write(1, "string\n", strlen("string\n")); instead of printf, since write bypasses buffering and directly writes to stdout. Does anyone know why this happens? What is the operating system's policy regarding this?

I am running multiple processes and want to exit with the appropriate exit code (this means error on failure, success otherwise) if any of them fails or exists. Additionally if any child process exits or fails, any other child processes should also be shut down. My current non-functional solution (yarn is just an example; may be any other command): #!/bin/bash -e # Run other process before start sh ./bin/optimize.sh trap 'exit' INT TERM trap 'kill -INT 0' EXIT # Run Schedule sh ./bin/schedule.sh & # Run a long running task yarn task & wait ./bin/schedule.sh: #!/bin/bash -e while true; do yarn schedule sleep 30 done If something in yarn schedule fails, everything exists correctly. But when I kill the process via ctrl+c or yarn task exits yarn schedule keeps running. How to get this working regardless of what the child processes are (bash, yarn, php or whatever)? I can't use GNU parallel.

There are various ways to access block devices over a network, like * nbd, alias network block device * iscsi, alias scsi over tcp * or simply we can mmap() files mounted in an nfs or cifs network share. I am now thinking about a process, mmap()-ing such block devices and then flock()-ing them or part of them. Obviously, as they are in a network share, such memory locks can happen concurrently on multiple devices. What I am trying to do, is *to distribute and handle these concurrent locks even across the network*, i.e. if a lock happens on a node, it should affect also other nodes. It probably requires some RPC-based protocol, which is likely not trivial, but possible to develop. Does such a thing exist?

I'm a bit confused about concurrency and parallelism in the bash shell. As I understand it, when we run commands in more than one subshells at the same time, these commands run in parallel on individual processor cores. **For example;**

cmd1 & cmd2 & cmd3 &

Here, the "ampersand" sign is run in the background (aka the subshells) of each command at the same time.It could be create in subshells in other ways. (Like writing in parentheses or using a pipe ..). In this direction, the questions I wonder the answers of; - Bash provides parallelism through subshells, other hand is concurrency also achieved using another method on bash? As far as I know, concurrency works in a way that a single CPU executes operations intermittently. Do I need to implement a method externally to achieve this, or is bash already working this way (concurrency ) anyway. - If I occupy all CPU cores using parallelism, will the system crash or is there a protection mechanism against this situation? - What is the difference between the parallel I provide with subshells and the GNU Parallel tool? If the GNU Parallel tool works better, how does it achieve this? - Which of the "Parallel" or "Concurrency" operations works more efficiently? - What kind of losses are experienced when making "parallel" or "concurrency" operation, unlike normal (sequential execution of commands)?

>Routines in both languages are inexpensive : goroutines are 2KB each, while Elixir processes are 0.5KB each. I understand that to start a process in BEAM requires 0.5KB of memory. This being so lightweight in the case of Elixir and Erlang, and to a lesser degree in the case of Go seems to be the advantage of using these runtimes when compared with other languages that rely on either their own not that inexpensive processes and threads or on the underlying OS 's processes and threads (that also supposedly require more memory). I want to know how much memory requires starting a process on a Linux-based system. I am aware that the memory usage depends on what the process is doing. But I assume that there is a memory cost of just starting a process that does nothing). What is that cost? Where can I read more about it? Are there any files / commands with which I can inspect this?

*This is not a "how to append and not overwrite" question. I'm not looking for a file that combines the output of two commands. It's just a mistake I made and I would like to understand why the system did what it did* I use a command (on a remote ssh command line) that takes a long time to complete and outputs data (line by line every few seconds) to stdout so I redirect it to a file: command > file.out & Sometimes the remote session disconnects but the command keeps on running in the background. I didn't know this so I run the same command again, *before the first one had finished*: command > file.out & When both processes have finished I would expect to have (after reading some answers at this site) a single file with the lines from both commands messed up, but the output file only has the output from one of the 2 executions. Why doesn't the file have both outputs intertwined (as warned in the comments here )? Which one of the 2 outputs does the final file belong to? **EDIT:** Removed one of the questions (why is the output file not locked for writing?) as it's explained here

I'm trying to find a way to lock a script based on a parameter given, but was unsuccessful in finding a proper answer. What I'm trying to achieve is prevent another user from running a script based on some parameter: so if user A executes the script with parameter JOHN_DOE (e.g: -d JOHN_DOE) and user B executes it with parameter -d ANNA_DOE then it runs without any problems, but if user B tries to execute it with JOHN_DOE as parameter while the first instance of the script hasn't finished running then it does not allow the user B to run it. Is there a proper way to achieve this?

### The setting ### Let's say I have an executable file, let's call it program, whose source code is unavailable (maybe proprietary/legacy). Each time this program is executed, it generates a file, let's call it file.txt, always with the same name and always on the same directory, depending on the command line argument passed to program. ### The problem ### I want to execute several instances of program concurrently. This may happen in a context in which program lives in a server that is listening for requests to execute program from the clients. The problem is that it seems impossible, as long as both instances would be writing to the same file.txt, and hence probably, this file will end up being corrupt. ### Possible solution ideas ### - Can I somehow redirect program's output to a file with a unique name (remember that the source is missing)? - Can I somehow "sandbox" program to make it behave as if executed on a separate file system (with negligible overhead)? - I have heard about LD_PRELOAD, but I don't know if it just works for overriding C standard library functions or if it also works for overriding Linux system calls in general. Any idea?

Let's say I have N bash commands stored in a file:

$ cat list_of_commands.txt
do_thing_1
do_thing_2
...
do_thing_N

The goal is to run them in parallel, but only 10 at a time (to avoid overloading the CPU). So, the first batch of commands (do_thing_1 to do_thing_10) starts running in parallel, and as soon as one of them are done, do_thing_11 should start running. When another command is done, do_thing_12 starts. Etc. Until all N commands are done. I've found the GNU parallel tool but after tinkering with it, I'm unsure if it's the right tool for the job. Thanks in advance!
~ Rijk

I want to execute four scripts at the same time with nohup. These scripts are all in the same folder. Would something like this work?:

nohup /.scrip1.sh & /.scrip2.sh & /.scrip3.sh & /.scrip4.sh

Would this print a single nohup.out?

#!/usr/bin/bash

TARGETS=(
"81.176.235.2"
"81.176.70.2" 
"78.41.109.7" 
)

myIPs=(
"185.164.100.1"
"185.164.100.2"
"185.164.100.3"
"185.164.100.4"
"185.164.100.5"
)

for t in "${TARGETS[@]}"
  do 
    for a in "${myIPs[@]}"
    do 
      echo "${a} ${t} -p 80" >>log 2>&1 &
      echo "${a} ${t} -p 443" >>log 2>&1 &
      wait
    done 
    
done

I want this code to start with echo commands for each IP in TARGETS executing them in parallel. At the same time the script is not meant to proceed with echo commands for more than one address in myIPs simulteously, hence I introduced wait in the internal loop. I want to have pairs of echo (each for the port 80 and 443) executed in parallel for each target in TARGETS. In other words I want to accomplish this (but sadly it does not work):

for t in "${TARGETS[@]}"
do &
  for a in "${myIPs[@]}"
  do 
    echo "${a} ${t} -p 80" >>log 2>&1 &
    echo "${a} ${t} -p 443" >>log 2>&1 &
    wait
  done 

done
wait

Yet, because it would increase my load averages too much, I do not want this: :

for t in "${TARGETS[@]}"
do
  for a in "${myIPs[@]}"
  do 
    echo "${a} ${t} -p 80" >>log 2>&1 &
    echo "${a} ${t} -p 443" >>log 2>&1 &
  done 
done
wait

How might I accomplish my objective? P.S. This is just a snippet of a more complex script. I wanted isolate the relevant issue, hence the use of echo instead of one of the networking commands.

I have a script which locks a file to avoid concurrent access to it, How can I execute this same script from two different terminals synchronously, to check if it works? Here is the script

#!/bin/bash

(
  flock -xn 200
  trap 'rm /tmp/test_lock.txt' 0
  RETVAL=$?
  if [ $RETVAL -eq 1 ]
  then
    echo $RETVAL
    echo  "file already removed"
    exit 1
  else
    echo "locked and removed"
  fi
) 200>/tmp/test_lock.txt

I have some files stored on a virtual machine that I'm downloading onto my PC. There are approximately 1 million files and I have been using the following command:

scp vm_user@IP:/home/vm_user/path_to_files /Users/documents

As you can imagine, this is slow as it downloads the files one by one. Are there quicker alternatives that can download the files asynchronously or apply concurrency to the downloads to increase download speed?

(this a dummy question)multiple processes are running in the background. my understanding is each CPU core can run only 1 process at a time. so someone has the ability to interrupt the current process(so the state of the process must be saved somewhere)(or the process itself does that) to run another one. when we have one core and in the current time a process is running, where is the other godfather process that watches these and handles the situation? what is the name of it? for example, we have two servers that are running on one PC with one core and listening on different ports. literally, this listening isn't a continuous job, the must be interrupted at least by another one, who handles this?

If I call some command, for instance an echo I can use the results from that command in several other commands with tee. Example: echo "Hello world!" | tee >(command1) >(command2) >(command3) With cat I can collect the results of several commands. Example: cat auxfile cat <(command1 < auxfile) \ <(command2 < auxfile) \ <(command3 < auxfile) fi } Readings and writings in auxfile seem to be overlapping, causing everything to explode.

I know that Linux OS's are typically multi-programmed, which means that multiple processes can be active at the same time. Can there be multiple kernels executing at the same time?

I have a bunch of files and for each row there is a unique value I'm trying to obscure with a hash. However there are 3M rows across the files and a rough calculation of the time needed to complete the process is hilariously long at 32days. for y in files*; do cat $y | while read z; do KEY=$(echo $z | awk '{ print $1 }' | tr -d '"') HASH=$(echo $KEY | sha1sum | awk '{ print $1 }') sed -i -e "s/$KEY/$HASH/g" $y done done To improve this processes speed I assume I'm going to have to introduce some concurrency. A hasty attempt based of https://unix.stackexchange.com/a/216475 led me to N=4 ( for y in gta*; do cat $y | while read z; do (i=i%N)); ((i++==0)); wait ((GTA=$(echo $z | awk '{ print $1 }' | tr -d '"') HASH=$(echo $GTA | sha1sum | awk '{ print $1 }') sed -i -e "s/$KEY/$HASH/g) & done done ) Which performs no better. Example input "2000000000" : ["200000", "2000000000"] "2000000001" : ["200000", "2000000001"] Example output "e8bb6adbb44a2f4c795da6986c8f008d05938fac" : ["200000", "e8bb6adbb44a2f4c795da6986c8f008d05938fac"] "aaac41fe0491d5855591b849453a58c206d424df" : ["200000", "aaac41fe0491d5855591b849453a58c206d424df"] Perhaps I should read the lines concurrently then perform the hash-replace on each line?

In The Linux Programming Interface, Chapter 60 talks about - designing a server to concurrently handle multiple clients, by using sockets, multiple processes/threads or thread/process pools - designing a server to be invoked by inetd which simplifies the sever program. If a server is to be used with inetd, it only has to deal with inputs from stdin and outputs to stdout, instead of sockets. inetd handles the part of using sockets, multiplex monitors for incoming requests, and for each incoming request, forks a process to execute a server program. I was wondering if a server program to be used with inetd can still have the same part of concurrently handling multiple clients, as a standalone version of the server program? Is it correct that inetd forks a new process for each incoming request, to execute the entire server program? So does the server program not need to handle multiple requests concurrently, but only one request? Does inetd makes the server program equivalent to a concurrent standalone version which forks a child process to process each received request? Is that a good choice compared to a standalone multithreaded server program? Thanks.

I'm working on a script in which I'm trying to have two concurrent timers within the same process. In the following example, I'm trying to update the ip address every 60 seconds, while outputting the ip every 5 seconds: while true; do ip=$(curl -4 -sf ifconfig.co) sleep 60 done & while true; do echo $ip sleep 5 done The problem here is that the first while loop creates a subprocess and thus does not share the ip address variable. But if I remove the &, the script will never get to the second loop. I could potentially integrate both loops into one, but I'm wondering if there isn't a better way to have these two loops run concurrently within the same process

The behavior of dash (and also bash) looks non-intuitive in this case. It seems that new_file is created before ls is executed. Is it specified in POSIX?