Sample Header Ad - 728x90

Running GNU Parallel on 2 or more nodes with Slurm scheduler

5 votes
2 answers
2032 views
I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up: Script#1: myCommands ./myscript --input infile.txt --setting 1 --output out1 ./myscript --input infile.txt --setting 2 --output out2 ./myscript --input infile.txt --setting 3 --output out3 ./myscript --input infile.txt --setting 4 --output out4 Script#2: run.sh #SBATCH --time=00:02:00 #SBATCH --nodes=2 #SBATCH --cpus-per-task=2 cat myCommands | parallel -j 4 This works, however it only uses one node. The two cores on that nodes are split into 4 threads to make room for 4 jobs as requested by parallel. That is not desirable. My searching indicates I will need a nodefile and a sshloginfile to accomplish this, but I see no examples online that work with Slurm, only with PBS system. How can I make the script (1) use both nodes, and (2) not split cores into threads?
Asked by cryptic0 (191 rep)
Jan 30, 2017, 06:24 PM
Last activity: Jun 15, 2025, 05:03 PM