I have created a shortcut in my KDE environment that performs the following command : bsub -q gui konsole -e tmux -Asmain It creates a konsole and automatically open a tmux session named main . The point is that when I activate this shortcut multiple time : sometimes it manages to reconnect to the existing main session, and sometimes it creates a new session named also main. Even though the machine selected by bsub is the same. Is there a way to enforce the usage of one single session created by tmux ?

Hi I guess this is a very basic question: Is there any specific syntax to do the "piping" when using "bsub"? I am asking that because when I had an issue like this: # try to run the same command successfully ran in another host bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimmed_file.fastq & but I met an error: fastq_quality_trimmer: input file (-) has unknown file format (not FASTA or FASTQ), first character = J (74) the second command seems not be able find out the output of the first command. # try to run command without using "|" bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq -o Trimmed_file.fastq # seemed to work! bsub fastq_quality_trimmer -Q33 -t 18 -l 20 -i Trimmed_file.fastq -o Trimme_file2.fastq & # Also seemed to work! # try to pipe again, did not work... bsub fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimme_file2.fastq & # when call command without bsub, it seemed to work. fastx_trimmer -Q33 -f 1 -l 230 -i myfile.fastq | fastq_quality_trimmer -Q33 -t 18 -l 20 -o Trimme_file2.fastq & # so the issue seemed to be "bsub" I would like to use bsub because my host is using gate node (the previous host has no gate node, so it does not matter whether I use "bsub" or not), and I do not want to create "traffic jam" by running command in the gate node. Any suggestion?

I'v lots bjobs running on lsf, jobs have two status, RUN and PEND. And I want to kill all bjobs with PEND status, how to do that use script? A hard-coded way I think is saving them in a file then parse every line to get the status and key. If the STAT is PEND then pass the key to bkill $key. But this is very complicated, is there any bkill function that can directly do this or a non hard-coded way to kill jobs with a specific status or name?

I have written

loop and parallelized it with

&

and limited the threshold to running

jobs at one time. Below is my script. I am reserving

cores

and

GB

memory through

command. The

I am running inside

loop requires

cores and

GB

memory. I am getting memory failure errors on some jobs. I think I am reserving only

cores and

GB

and trying to run

jobs at a time which might be causing memory failure errors on some jobs. My question is how do I parallelize such that all

jobs are using the same amount of cores and memory. I submitted using the command

< example.sh

#!/bin/bash #BSUB -J cnt_job # LSF job name #BSUB -o cnt_job.%J.out # Name of the job output file #BSUB -e cnt_job.%J.error # Name of the job error file #BSUB -n 32 # 32 cores #BSUB -M 262144 # 256 GB #BSUB -R "span[hosts=1] rusage [mem=262144]" n=0 maxjobs=3 for sample in $main ; do for run in $nested ; do sample_pipe count --id="$run_name" \ --localcores=32 \ --localmem=256 & done cd .. # limit jobs if (( $(($((++n)) % $maxjobs)) == 0 )) ; then wait # wait until all have finished echo $n wait fi done

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to *M* jobs either running or pending at any one time, of which at most *L* *M*? Or *N* ≫ *M* even? --- The *N* ≫ *M* scenario occurs *extremely* often in my line of work, so often in fact that a hope to find tools to facilitate dealing with it would not be unreasonable1. One very general and straightforward way to get around the scheduler-imposed limits is to split the *N* independent commands into *L* separate one-time "batching" scripts, and submit the latter to the scheduler, as *L* separate jobs2. Granted, creating such one-time batching scripts is a dull, somewhat annoying chore, but someone who is handy with their shell, or with a scripting language like Python, Perl, etc., can easily take care of it, and even home-roll their own hacks to automate it. My question is, however, are there publicly (and freely) available tools in the Unix ecosystem that can be used even by those with less programming skill to automate the chore of generating *L* such batching scripts, given as input a list of *N* independent commands? --- 1Actually, the scenario occurs so often that I am surprised that schedulers do not already have built-in support for it. At least the schedulers that I am most familiar with (Slurm and LSF) do not have any such support, as far as I can tell. Please correct me if I missed something. 2 More generally, one could to batch the *N* commands into *k* batching scripts, as long as *k* ≤ *M*, but, in my experience, choosing *k* = *L* is the most straightforward way to achieve a maximal, or near-maximal, throughput under these constraints. The reasons for this are not too difficult to see, but an adequate discussion of the matter would require more time than I want to take up here.

Every day, I encounter at least one Wikipedia article about something computer-related, which should logically make sense to me if I've spent my entire life growing up with and dealing with computers on a technical/deeper level, never playing any games or using them in a "consumer" manner. But then I see one of these Wikipedia articles about some cryptic nonsense and I have no idea what it's saying. It's like reading a text written by aliens from a different on not just a different planet, but a completely different universe: https://en.wikipedia.org/wiki/Platform_LSF > Platform Load Sharing Facility (or simply LSF) is a workload management platform, job scheduler, for distributed high performance computing. It can be used to execute batch jobs on networked Unix and Windows systems on many different architectures. > IBM Platform Application Center: Web interfaces for job submission, management and remote visualization. > IBM Platform RTM: A real-time dashboard for monitoring global workloads and resource. > IBM Platform License Scheduler: License management tool with policy-driven allocation and tracking of software licenses. > IBM Platform Analytic : Analytic tool for visualizing and analyzing workload data. > IBM Platform Process Manager: An interface for designing complex engineering computational processes > IBM Platform Session Scheduler: Scheduling for LSF > IBM Platform Dynamic Cluster: Cloud management software to change static cluster into dynamic share cloud resources ... what? This is literal word salad to me. They might as well have randomly generated this article. Is this an actual product of some kind? That is actually used? That has a purpose? What is a "workload management platform, job scheduler, for distributed high performance computing"? Why would you need some incredibly cryptic and weird separate product which is somehow related to IBM but also open source (???) to run commands in a batch script? When reading that article, and many like it, and many other websites as well (which apparently sell some service or software), I don't feel like I belong in this world, or that I grew up with computers. I feel like somebody who has time-traveled from the distant, ancient past to a confusing, scary future where humans no longer resemble humans at all, and there is such a degree of abstraction and convoluted software that it might as well be magic at this point, and few if anyone can tell the difference between randomly generated texts and actual product descriptions. > It can be used to execute batch jobs on networked Unix and Windows systems on many different architectures. Again, why is this a separate product/thing? What does that sentence even mean? I know what a batch job is, I know what Unix and Windows are, I know what networked means, and I know what architectures refer to, but I don't understand what that sentence is saying. I'm quite serious about this question: What in the hell is this?

So this is a super weird issue. I have a python script that calls many bash commands via subprocess.call based on certain criteria. Now, the script runs just fine manually but when thrown into a cronjob it fails, BUT only when it gets to a certain part of the code. This part of the code runs a

and a

command on a user. I've tried using subprocess.call, subprocess.Popen, subprocess.check_output for these two commands and every time it reaches them, it hangs and does nothing. I then get this message in var/spool/mail/root

File "/root/Desktop/script.py", line 75, in 
    print subprocess.check_output(['bstat' '-q' 'viz' '-u' ,user,])
  File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I've tried: using absolute paths for EVERY command possible, changing directories to the directory of the script before running, calling /bin/python before running. I'm at a total loss. What's even more weird, is there are other subprocess.call commands that work just fine when calling a bash script but when it comes to those two commands it doesn't know what to do. Below is the first subprocess command that it hangs on:

subprocess.check_output(['bstat' '-q' 'viz' '-u' ,user,])

When using LSF command bjobs, I would like to get instantly changing output if I submit another job, because I feel stressful to run the same command again and again. I would like something like top refreshing the output of the list of processes. In top that is not needed, it autorefreshes again and again. I would like to auto-refresh the output of the bjobs command automatically.

I'm writing a script that collect job information. I would like to get the wall time limit for each pending job (as specified with bsub -W) using bjobs. I can get information such as the number of cpus using a command like: bjobs -u all -p -o 'jobid user min_req_proc' But I can't find any keyword that will give me wall time. Does such a method exist? Right now I have to run bjobs -l for every job to get that information, which is very slow.

I have an LSF queue, say it's queue A. I only want higher priority jobs in queue A to preempt lower priority jobs in queue A, but not jobs in any other queues. If I set queue A to be preemptable, could I set it to be preemptable only by itself (queue A)? Or if this is not allowed, what could achieve my goal?

I have somewhat maybe little bit strange question running my jobs in my ubuntu laptop instead of running them hpc clusters. The reason I want to to that is now I have 6 core 12 threads. I used to run my jobs on hpc cluster of my previous university. What I was doing, 1) connecting to hpc cluster and go to where the execution file is which is wave_func.out submitting the jobs to lsf 3) bsub -q linux22 -i ./w1.in -o ./w1.out ./wave_func.out Then the if there is available thread in 'redhat7_211' my simulation starts. My question is, How can I perform these simulation on my personal ubuntu OS pc ?

We have some shell scripts that presently utilize Platform LSF to manage job execution. These scripts will eventually be migrated to an environment that uses a different job scheduler. During the migration phase it would be ideal to have the same script support job scheduling in both environments, so if it detects that the LSF environment is present, it uses LSF-specific instructions, else it uses the commands relevant to the other environment. I've thought of a few possible solutions: - ***Check for LSF-specific environment variables*** e.g. $LSF_BINDIR, $LSF_LIBDIR, $LSF_SERVERDIR if [[ -n $LSF_BINDIR ]]; then # Yes LSF else # No LSF fi - ***Check for the availability of LSF-specific commands*** e.g. which bsub, which bhosts, which bjobs if [[ -z $( which bsub 2>&1 | grep "/usr/bin/which" ) ]]; then # LSF present else # LSF absent fi I don't know if these checks are sufficient, or whether there is a better (as in less hacky), more reliable method for detecting the presence of Platform LSF on a system.

I am currently working with the bsub job submission system from Platform LSF. I need to submit job scripts, but I am running into a problem with passing arguments to the job script. What I need is as follows: the job script is a bash script that takes one command line argument. I need to pass the argument to that script, then redirect the script to be used as input to bsub. The problem is that the argument isn't being applied to the script. I have tried: bsub < script.bsub 1 script.bsub is my script and 1 is the numeric argument. This approach doesn't work. The 1 is being treated as an argument to bsub. Another approach: bsub < "script.bsub 1" This approach makes bsub treat the entire double-quoted line as a filename. Is there a way to apply a command line argument to a script first, then redirect that script with the argument into another program as input? If anyone has experience with bsub that would be even better. UPDATE AND WORKAROUND: It seems applying a command line argument to a bsub file is a very complicated process. I tried the HEREDOC method stated below, but bsub acted as if the script filename was a command. So the easiest way to get around this problem is just to not use input redirection at all. To pass an argument to a script to be run in bsub, first specify all bsub arguments in the command line rather than in the script file. Then to run the script file, use "sh script.sh [arg]" after all of the bsub arguments. Thus the entire line will look something like: bsub -q [queue] -J "[name]" -W 00:10 [other bsub args] "sh script.sh [script args]" This doesn't answer my original question, but at least with bsub this is the simplest resolution to my problem.

I want to monitor LSF job count through command prompt. %j will not give me the LSF job count. Can anyone tell how to configure command prompt to show the LSF job count?

What is the correct way to specify how long a job can run on a computer cluster using LSF as the resource manager/ job scheduler? LSF is a job scheduler that computing clusters use to allocate resources and decide when a job that users submit gets run and how it gets run. Based on IBM's knowledge base , can I do the following if I needed 5 days? busb -We 120:00