Sample Header Ad - 728x90

GNU Parallel Python Semaphore

1 vote
1 answer
427 views
I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I open this file in each parallelized thread, and the threads are stepping on eachother's toes while writing. I want to implement a FIFO semaphore, which I presume would have to be in the GNU parallel script, however I am unsure how to do that if I am accessing the file from within Python. My current script is:
#!/bin/bash
time parallel -j$(nproc) -N0 python3 ./polynomial_generator.py ::: {1..10}  --progress echo {} >/tmp/out
Asked by Alex Williams (65 rep)
Jan 20, 2020, 07:30 PM
Last activity: Jan 21, 2020, 01:20 PM