Tools for batching N commands over L scripts (for N≫L)?
1
vote
1
answer
53
views
Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to *M* jobs either running or pending at any one time, of which at most *L* *M*? Or *N* ≫ *M* even?
---
The *N* ≫ *M* scenario occurs *extremely* often in my line of work, so often in fact that a hope to find tools to facilitate dealing with it would not be unreasonable1.
One very general and straightforward way to get around the scheduler-imposed limits is to split the *N* independent commands into *L* separate one-time "batching" scripts, and submit the latter to the scheduler, as *L* separate jobs2.
Granted, creating such one-time batching scripts is a dull, somewhat annoying chore, but someone who is handy with their shell, or with a scripting language like Python, Perl, etc., can easily take care of it, and even home-roll their own hacks to automate it.
My question is, however, are there publicly (and freely) available tools in the Unix ecosystem that can be used even by those with less programming skill to automate the chore of generating *L* such batching scripts, given as input a list of *N* independent commands?
---
1Actually, the scenario occurs so often that I am surprised that schedulers do not already have built-in support for it. At least the schedulers that I am most familiar with (Slurm and LSF) do not have any such support, as far as I can tell. Please correct me if I missed something.
2 More generally, one could to batch the *N* commands into *k* batching scripts, as long as *k* ≤ *M*, but, in my experience, choosing *k* = *L* is the most straightforward way to achieve a maximal, or near-maximal, throughput under these constraints. The reasons for this are not too difficult to see, but an adequate discussion of the matter would require more time than I want to take up here.
Asked by kjo
(16299 rep)
Aug 15, 2021, 02:58 PM
Last activity: Aug 17, 2021, 06:33 PM
Last activity: Aug 17, 2021, 06:33 PM