I've read that the kernel implements synchronisation mechanisms when accessing files. For example, if we try and write or read to a file in the file system using read() or write() from different processes at the same time, the kernel will prevent race conditions. How exactly is this implemented? I have used Mutexes and Semaphores when writing code before which prevents different threads or processes executing a certain part of the code at the same time. In this case, I assume that the kernel should only implement a locking mechanism when more than one process or thread tries to read or write to the same file descriptor, not any time read() or write() is called, which could be for any file descriptor. How would this be achieved?

I am trying to work on some parallelization of many processes (task send/to be executed on many (let's say hundreds) nodes). I came across this solution: https://unix.stackexchange.com/a/216475

-bash
    # initialize a semaphore with a given number of tokens
    open_sem(){
        mkfifo pipe-$$
        exec 3pipe-$$
        rm pipe-$$
        local i=$1
        for((;i>0;i--)); do
            printf %s 000 >&3
        done
    }
    
    # run the given command asynchronously and pop/push tokens
    run_with_lock(){
        local x
        # this read waits until there is something to read
        read -u 3 -n 3 x && ((0==x)) || exit $x
        (
         ( "$@"; )
        # push the return code of the command to the semaphore
        printf '%.3d' $? >&3
        )&
    }
    
    N=4
    open_sem $N
    for thing in {a..g}; do
        run_with_lock task $thing
    done

I need some explanation here: **open_sem()** 1. what does exec 3pipe-$$ do? 2. why is it removed afterwards? **run_with_lock()** 1. what does this && ((0==x)) || exit $x part mean? 2. ( "$@"; ) - as far as i know this is list of all arguments passed... but what does it do here? These are main obstacles for me to understand the process, but feel free to explain the whole flow :) PS: I would just make a comment under that post, but I have just registered and do not have reputation to do so. Also it might be useful for others. Thanks! J.

In a shell script I can use: ipcmk -S 4 To create a semaphore with 4 slots. With ipcrm I could delete the semaphore again. But how can I use it? I cannot find any semwait, ipcsemget or similar. How is this done?

Obviously O_CREAT and O_EXCL are not required when opening an existing semaphore. O_CREAT is required when creating a new semaphore. O_EXCL is only meaningful when OR-ing with O_CREAT, specifying that if a semaphore with the given name already exists, then an error is returned. Linux manual page for sem_open said that > Definitions of the flags values can be obtained by including but I did not find any flag in fcntl.h that told me how to open an existing semaphore.

I've been greatly inspired by this question: https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop to parallelise some tools I've written that involve very loooong while read loops (ie doing the same task / set of tasks across paths given in an input file. The input file would contain around 90,000 lines, and growing). I've done all the work to 'shoehorn' PSkocik's 'N processes with a FIFO-based semaphore' example into my code...

-bash
# initialize a semaphore with a given number of tokens
open_sem(){
    mkfifo /tmp/pipe-$$
    exec 3/tmp/pipe-$$
    rm /tmp/pipe-$$
    local i=$1
    for((;i>0;i--)); do
        printf %s 000 >&3
    done
}

# run the given command asynchronously and pop/push tokens
run_with_lock(){
    local x
    # this read waits until there is something to read
    read -u 3 -n 3 x && ((0==x)) || exit $x
    (
     ( "$@"; )
    # push the return code of the command to the semaphore
    printf '%.3d' $? >&3
    )&
}

N=4
open_sem $N
for thing in {a..g}; do
    run_with_lock task $thing
done

However, my 'old' code had a nice progress counter built into the read loop (code below is abbreviated) and yes, I know there is a weird mix of echo, awk and printf I tend to reuse code from other scripts I've written where maybe some of the code was based off of other online examples etc... I'm sure I can tidy this up... but its works and I'm the only one using this code!:

-bash
## $temp1 is the file with 90,000 lines to read over
## $YELLOW is a global variable exported from my bashrc with the escape code for yellow text
## $GREEN is a global variable exported from my bashrc with the escape code for green text
## $CL is a global variable exported from my bashrc with the escape code for Clear Line
## $NC is a global variable exported from my bashrc with the escape code to revert text colour back to normal

num_lines="$(cat $temp1 | wc -l)"
percent_per_line="$(awk "BEGIN {print 100/$num_lines}")"
progress_percent='0'
current_line='1'

echo -ne "${CL}${YELLOW}PROGRESS: ${progress_percent}% ${NC}\r"
while read line; do
    ############################################
    ##commands to process $line data as needed##
    ############################################

    progress_percent="$(awk "BEGIN {print $progress_percent + $percent_per_line }")"
    awk -v y=$YELLOW -v nc=$NC -v progress=$progress_percent -v current_line=$current_line -v total_lines=$num_lines 'BEGIN {printf (y"\033[2KPROGRESS: %.3f%%   (%d OF %d)\n\033[1A"nc, progress, current_line, total_lines) }' 
    #I think I mixed my global var escape codes with typed out ones cause I was I forgot / have no need to export \033[2K and \033[1A for anything else?
    ((current_line++))
done < "$temp1"

echo -e "${CL}${GREEN}PROGRESS: 100.000%   (${num_lines} OF ${num_lines})${NC}"

I'm trying to find a way to again, shoehorn something with a similar output into the 'new' FIFO-semaphore code.... I just can't work out how! does it go into the run_with_lock function, if so where, and I would need to pass into that function the percent_per_line and num_lines variables but it passes $@ within it :( I feel I'm not fully understanding how the FIFO-semaphore works as if I did I would use another semaphore message type thing to pass the data I need around... Any help would be massively appreciated as it helps me learn and improve!!

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I open this file in each parallelized thread, and the threads are stepping on eachother's toes while writing. I want to implement a FIFO semaphore, which I presume would have to be in the GNU parallel script, however I am unsure how to do that if I am accessing the file from within Python. My current script is:

#!/bin/bash
time parallel -j$(nproc) -N0 python3 ./polynomial_generator.py ::: {1..10}  --progress echo {} >/tmp/out

I try to improve with semaphores some photo processing script I use. I run sable GNU/DEBIAN There is a function called travail that uses a given filename as an argument, and that processes the file. For now, that function is called in the same script with a loop : for i in *.png ; do travail $i & done The problem is that it uses a lot of memory if there are a lot of pictures. I would like to use semaphores to limit the number of threads. I tried: for i in *.png ; do sem -j+0 travail $i done sem --wait But it does not work, with the following message : /bin/bash: travail : commande introuvable (that means command not found) I try to add export -f travail after the function has been defined, but it does not work. Exemple of the output (2 files for this exemple) > travail sur image 113_XT1S3739.png convert-im6.q16: unable to open > image `x': Aucun fichier ou dossier de ce type @ > error/blob.c/OpenBlob/2874. convert-im6.q16: no decode delegate for > this image format `' @ error/constitute.c/ReadImage/560. > convert-im6.q16: invalid argument for option `-quality': -unsharp @ > error/convert.c/ConvertImageCommand/2460. composite-im6.q16: invalid > argument for option `-quality': > sortie/grand_format/ne_pas_publier_113_XT1S3739.jpg @ > error/composite.c/CompositeImageCommand/1241. travail sur image > 113_XT1S3779.png convert-im6.q16: unable to open image `x': Aucun > fichier ou dossier de ce type @ error/blob.c/OpenBlob/2874. > convert-im6.q16: no decode delegate for this image format `' @ > error/constitute.c/ReadImage/560. convert-im6.q16: invalid argument > for option `-quality': -unsharp @ > error/convert.c/ConvertImageCommand/2460. composite-im6.q16: invalid > argument for option `-quality': > sortie/grand_format/ne_pas_publier_113_XT1S3779.jpg @ > error/composite.c/CompositeImageCommand/1241. etc. For more information, Here is the script # !/bin/bash # _____________________________ Paramètres T_WEB="1000" # Taille maxi des images pour le web Q_WEB="100" # Qualité des images pour le web (100 car il faut les réouvrir pour mettre le logo) # H_MOY="1795" # Hauteur maxi des images pour impression ?x15 L_MOY_11="1347" # Largeur maxi des images pour impression 11x15 L_MOY_10="1204" # Largeur maxi des images pour impression 10x15 Q_MOY="96" # Qualité des images pour impression 10x15 # Q_GRA="99" # Qualité des grandes images # B_INT="0.1%" # Taille de la bordure intérieure B_CEN="0.3%" # Taille de la bordure intérieure B_EXT="4%" # Taille de la bordure extérieure C_INT="white" # Couleur de la bordure intérieure C_CEN="black" # Couleur de la bordure intérieure C_EXT="white" # Couleur de la bordure extérieure # # __________________________________________________________________ # # Fonction principale # function travail { # Lecture du nom de fichier à traiter local f=$1 echo 'travail sur image' $f # Récupération d'infos sur l'image en cours local LARG=convert -quiet -ping $f -format "%w" info: local HAUT=convert -quiet -ping $f -format "%h" info: local LARG=${LARG:0:4} # On ne conserve que les 4 premiers chiffres, le reste n'est pas bon local HAUT=${HAUT:0:4} local DECALE_X=$(($LARG/24)) local DECALE_Y=$(($HAUT/22)) # # Récupération du nom du fichier et de l'extension local nomfichier="${f%%.*}" # # Recuperation donnees EXIF (elles sont écrites dans un fichier temporaire) if [ -f "${nomfichier}.RAF" ] then exiv2 -q ex ${nomfichier}.RAF fi if [ -f "${nomfichier}.NEF" ] then exiv2 -q ex ${nomfichier}.NEF fi # # Ajout des bordures convert $f -quiet -bordercolor ${C_INT} -border ${B_INT} -bordercolor ${C_CEN} -border ${B_CEN} -bordercolor ${C_EXT} -border ${B_EXT}x${B_EXT} -quality 01 /tmp/$f # # Création image WEB convert /tmp/$f -resize ${T_WEB}x${T_WEB} -quality ${Q_WEB} -unsharp 3 sortie/web/${nomfichier}_pour_internet.jpg if [ -f "${nomfichier}.NEF" ] then mv ${nomfichier}.exv sortie/web/${nomfichier}_pour_internet.exv exiv2 in sortie/web/${nomfichier}_pour_internet.jpg exiftool -Orientation=1 -n sortie/web/${nomfichier}_pour_internet.jpg rm sortie/web/${nomfichier}_pour_internet.jpg_original fi # # Ajout logo puis conversion en JPG composite -compose Over -geometry +$DECALE_X+$DECALE_Y -gravity SouthEast /home/pierre/Documents/Photo/baniere-logo/logo_800x600.png /tmp/$f -quality ${Q_GRA} sortie/grand_format/ne_pas_publier_${nomfichier}.jpg # if [ -f "${nomfichier}.NEF" ] then mv sortie/web/${nomfichier}_pour_internet.exv sortie/grand_format/ne_pas_publier_${nomfichier}.exv exiv2 in sortie/grand_format/ne_pas_publier_${nomfichier}.jpg exiftool -Orientation=1 -n sortie/grand_format/ne_pas_publier_${nomfichier}.jpg rm sortie/grand_format/ne_pas_publier_${nomfichier}.jpg_original rm sortie/grand_format/ne_pas_publier_${nomfichier}.exv fi # Suppression des fichiers inutiles rm /tmp/$f if [ -f "${nomfichier}.RAF" ] then rm ${nomfichier}.exv fi } # # _____________________________ Création des dossiers et préparation ___________________________________________ if test -e sortie; then rm -rf sortie fi mkdir sortie mkdir sortie/web mkdir sortie/grand_format # # _____________________________ Appel de la fonction en parallèle sur toutes les images PNG # # for i in *.png ; do travail $i & done wait # # _____________________________ Fin de la fonction ________________________________________________________ # echo echo TERMINÉ

If I have a process that is in sleep state and I can see in ps column WCHAN that is waiting to obtain a semaphore, is there any way to find the address of that semaphore?

#proc1.sh #!/bin/sh touch /tmp/proc1.signature.mutex #do something for long time sleep 100 rm -rf /tmp/proc1.signature.mutex #proc2.sh #!/bin/sh touch /tmp/proc2.signature.mutex #do something for long time sleep 100 rm -rf /tmp/proc2.signature.mutex #proc3.sh #!/bin/sh touch /tmp/proc3.signature.mutex #do something for long time sleep 100 rm -rf /tmp/proc3.signature.mutex #core.sh Now is there a way to wait for deleting /tmp/proc[?][*].signature.mutex all such file using loop or something and then continue further execution How to achieve objective of core.sh

A named semaphore (using semaphore.h) is identified by a name in the form /somename; that is, a null-terminated string of up to NAME_MAX-4 (i.e., 251) characters consisting of an initial slash, followed by one or more characters, none of which are slashes. As the name corresponds to pathname in filesystem. Where is this semaphore located?ipcs is for System V semaphores.How to locate POSIX semaphores?

After going through the bash documentation , this question and this one it's still not clear to me how can I perform atomic write (append) operations to a file in bash. I have a script that is run in multiple instances and at some point must write data to a file: echo "$RESULT" >> pwd/$TEMP_DIR/$OUT_FILE How is it possible to make all write operations from all concurrently running scripts to that file atomic (so that data from one instance doesn't overlap data from another)?

if binary semaphore can protect a resource atomically, then what is the benefit of mutex. For example, sem_init(&sem, 0, 1); sem_wait(&sem); critical session; sem_post(&sem); Please clarify any benefits of mutex over semaphore.

Not sure if the question is pertinent for this forum. We have a server with 3 apache instances running with the "same" app/functional ID These instances communicate with their corresponding Siteminder webagents using inter process communication on the same box. Now when I execute ipcs command to look for semaphores and sharedmem objects, it displays "all" the objects and I am not sure which ones to remove using the ipcrm command. Is there a way to display just the ones that are being used by a specific apache instance? I know a better option would have been to have 3 different app ids for the apache instances than 1.

In order to use unnamed POSIX semaphores with multiple processes one has to create/obtain access to a portion of shared memory. After that, one process has to initialize the semaphore by calling sem_init with a reference to the shared memory as first parameter. However, this is only allowed to be done once! [POSIX](http://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_init.html) says: > Attempting to initialize an already initialized semaphore results in undefined behavior. The problem is that there is no way to test if the semaphore already exists and is initialized, and to initialize it if not, in an atomic way. One might be tempted to simply add a boolean flag to the shared memory that indicates if the semaphore was already initialized but that obviously produces a race condition - something one definitely does not want when working with semaphores :) XSI (aka System V) semaphores require some rather convoluted way to initialize properly without races but it can be made safe (cf. the implementation in APUE for example). Also, named POSIX semaphores do not show this problem either since sem_open accepts the O_EXCL flag and can initialize the value of the semaphore as well. So, how is one supposed to share unnamed POSIX semaphores between processes safely without relying on other ways to synchronize or forking the other processes only after initializing the semaphore? What do I miss?

I wonder how to get processes currently running semaphores by /proc ? I guess it's possible by SysVIPC subdirectory.But I don't know how to use this commands. Ubuntu 12.10

I understand that Spinlocks are real waste in Linux Kernel Design. I would like to know why is it like spin locks are good choices in Linux Kernel Design instead of something more common in userland code, such as semaphore or mutex?

What are the basic differences between spin locks and semaphores in action?

I have an Apache 1.3 on a Sun machine (SunOS 5.10 Generic_138888-01 sun4u sparc SUNW,Sun-Fire-15000). If I keep the following two lines in httpd.conf, the server will not start: LoadModule sm_module "/cmsm_sw/epiapr1/netegrity/webagent/bin/mod_sm.so"
SmInitFile "/apache/conf/WebAgent.conf" This is all I get when I try to start Apache: :/apache/bin # ./apachectl start
1392: CSemCounter created a semaphore with ID 1157627995 using key 989855746. The mod_sm.so is the netegrity siteminder module (iPlanet). Today there were some network issue (which were solved in the mean time) so I assume that this might have something to do. Maybe I should restart some services? I appreciate any help/hints.

I have the following shell script (one liner), which I wanted to use to identify directories that have the exact same contents. I'm using it, to identify and remove duplicate (child) directories. When I try to run the same script with sem, I encounter No such file or directory errors. **Example - no parallel threads** find -type d -links 2 | while read i; do \ find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \ | sort | md5sum | xargs -I {} echo {} $i ; \ done Gives me: e94d32e2a683d46d49c7580d649f7888 - ./Daft Punk/Alive 2007 2 e94d32e2a683d46d49c7580d649f7888 - ./Daft Punk/Alive 2007 --- **Example - using sem** find -type d -links 2 | while read i; do sem -j+0 \ find "$i" -type f -print0 | xargs -r0 md5sum | awk '{ print $1 }' \ | sort | md5sum | xargs -I {} echo {} $i ; \ done; sem --wait Gives me: find: `./Daft': No such file or directory find: `Punk/Alive': No such file or directory find: `2007': No such file or directory find: `2': No such file or directory d41d8cd98f00b204e9800998ecf8427e - ./Daft Punk/Alive 2007 2 find: `./Daft': No such file or directory find: `Punk/Alive': No such file or directory find: `2007': No such file or directory d41d8cd98f00b204e9800998ecf8427e - ./Daft Punk/Alive 2007 ### Questions: 1. Why the difference in behaviour? 1. How do I remove/fix the No such file or directory from the sem script? 1. Are there any other improvements I could make in the script? (there is lots of awk and xargs)

I want to run multiple Bash shell scripts in parallel. However, I want to avoid race conditions. What Unix commands are *truly* atomic that I could use for this purpose, and how can I use them?