Submitting HPC jobs within an HPC job

0 votes
1 answer
129 views
                          I have a large script which relies on input arguments (with getopts). One of these arguments is a directory containing files (all named *bam) This script has 2 parts: 

 - Part1: based on input *bam files, calculate one specific number. To be clear, the result is one single number, NOT one number per file.
 - Part 2: using the number found in part1, perform a series of operations on each *bam file.

Now, originally, part1 was very quick, computationally speaking. So my setup was:

 - Run script on terminal: bash script.sh
 - Within the script.sh, for part2, make a HPC job submission for each file

However, now that I need to analyze many more files than originally planned, I am realising that Part1 will also be computationally heavy - I therefore need to also run this on the HPC.

So my question is:

 - Is it possible to submit an HPC job which submits jobs in it?
 - In other words, can I submit script.sh as a job and and still have it submit jobs in its part2?

To be clear, here is an example of what my script might look like: 

    #!/usr/bin/bash

    # PART 0: accept all input arguments

    USAGE() { echo "Usage: bash $0 [-b ] [-o ] [-c ]" 1>&2; exit 1; }

    if (($# == 0)); then
            USAGE
    fi

    # Use getopts to accept each argument

    while getopts ":b:o:c:h" opt
    do
        case $opt in
           b ) BAMFILES=$OPTARG
            ;;
           o ) OUTDIR=$OPTARG
            ;;
           c ) CHROMLEN=$OPTARG
            ;;
           h ) USAGE
            ;;
           \? ) echo "Invalid option: -$OPTARG exiting" >&2
            exit
            ;;
           : ) echo "Option -$OPTARG requires an argument" >&2
            exit
            ;;
            esac
        done

    # PART1: calculate this unique number
    
    NUMBER=0    

    for i in  $(ls $BAMFILES/*.bam)
    do
      make some calculations on each file to obtain a number ...
      keep only the smallest found number and assign its value to $NUMBER
    done

    echo "Final number is ${NUMBER} "

    # PART2: Using $NUMBER that we found above, submit a job for each *bam file

    for i in $(ls $BAMFILES/*bam)
    do
    
        if [ ! -f ${OUTDIR}/${SAMPLE}.bw ];
        then 
            command=" command -options -b $NUMBER $i"

        echo $command | qsub -V -cwd -o $OUTDIR -e $OUTDIR -l tmem=6G -l h_vmem=6G -l h_rt=3600 -N result_${SAMPLE}

        fi

    done





                        
Asked by mf94 (229 rep)
Aug 21, 2018, 04:55 PM
Last activity: Aug 21, 2018, 11:27 PM
Submitting HPC jobs within an HPC job

Related Questions
