What’s the right way to pull a _complete_ answer to an InfluxQL query over http? I’m using the [acct_gather plugin](https://slurm.schedmd.com/acct_gather.conf.html) for a slurm cluster. It sends resource usage data to an [influxdb v1](https://docs.influxdata.com/influxdb/v1/) database. So if I write

#SBATCH --profile=Task

in an sbatch file, it records things like memory, I/O, and CPU usage to the database. But if I try to ask for that data as a json file, e.g.,...

jobid=12345
curl -G 'http://:/query ?' \
  --data-urlencode "db=myslurmdatabase" \
  --data-urlencode 'q=select "value" from /./ where "job"='"'$jobid'"

...then I get a **partial** response with only one type of measurement ("CPUFrequency"):

{
  "results": [
    {
      "statement_id": 0,
      "series": [
        {
          "name": "CPUFrequency",
          "columns": [
            "time",
            "value"
          ],
          "values": [

...

          ],
          "partial": true
        }
      ]
    }
  ]
}

I think this happens for jobs that have run past a certain number of data points. ## What I've found - In [this thread on github](https://github.com/grafana/grafana/issues/7380) somebody asked: > So how does it work? Do you get a url with the second chunk or does the http response contain multiple json bodies? Is this compatible with the json decoders in browser? People replied to the effect that modern browsers can handle it, but I don’t think they answered the question directly. - [There’s a “chunked” parameter](https://docs.influxdata.com/influxdb/v1/tools/api/#query-http-endpoint) for the /query endpoint. The options are either true (in which case it chunks based on series or 10,000 data points), or a specific number of points (in which case it chunks based on that number). Chunking happens either way. But it’s not clear to me how to get the _next_ chunk. - It looks like somebody has written [a third-party program](https://github.com/matthewdowney/influxdb-stream) that can stream the chunked results from a query. But is it possible with curl, or would I have to use something like this?

I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up: Script#1: myCommands ./myscript --input infile.txt --setting 1 --output out1 ./myscript --input infile.txt --setting 2 --output out2 ./myscript --input infile.txt --setting 3 --output out3 ./myscript --input infile.txt --setting 4 --output out4 Script#2: run.sh #SBATCH --time=00:02:00 #SBATCH --nodes=2 #SBATCH --cpus-per-task=2 cat myCommands | parallel -j 4 This works, however it only uses one node. The two cores on that nodes are split into 4 threads to make room for 4 jobs as requested by parallel. That is not desirable. My searching indicates I will need a nodefile and a sshloginfile to accomplish this, but I see no examples online that work with Slurm, only with PBS system. How can I make the script (1) use both nodes, and (2) not split cores into threads?

If I run a Slurm job with sbatch, [I can specify output and error filenames](https://unix.stackexchange.com/questions/285690/slurm-custom-standard-output-name?rq=1) with a [custom format](https://slurm.schedmd.com/sbatch.html#lbAH) . But how can I **look up** these filenames, given the job ID (e.g. 123456) of a running job? For example:

sbatch --wrap="bash -c 'echo stderr here >&2; echo stdout here; sleep 60'" \
  --error="slurm_outputs/stderr_%j_%N_%x.out" \
  --output="slurm_outputs/stdout_%j_%N_%x.out"

This uses the placeholders - %j for the job ID (123456), - %N for the name of the node it's running on (node1), - %x for the job name (wrap), so we end up with files stderr_123456_node1_wrap.out and stdout_123456_node1_wrap.out under slurm_outputs. **Near-solution:** I can get a json from squeue, which *almost* gives me what I want:

$ squeue --jobs=123456 --json |
    jq '.jobs[].standard_output, .jobs[].standard_error'
"/path/to/slurm_outputs/stdout_123456_%N_wrap.out"
"/path/to/slurm_outputs/stderr_123456_%N_wrap.out"

But the %N string is left uninterpreted—which I imagine is because the job scheduler doesn't know what node it's going to use before it allocates one, and squeue just has this old information. So I can’t immediately do something like

tail -f "$(squeue --jobs=123456 --json |
             jq --raw-output '.jobs[].standard_output')"

to follow the standard output of the job I just submitted. But that would be nice! Does slurm provide a way to look up the “finished” filename (i.e., with all placeholders filled)?

We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this in sbatch using --export to pass in the variable but can't get the variable to be interpolated for the std error/output file names. I think it's working for the job name (-J) and --wrap parts though. for i in *fastq.gz; do sbatch \ --export=i=$i --error='$i.eo%j' --output='$i.eo%j' \ -J trim_basename ${i} \ --wrap="fastq_trim_single.sh ${i}" \ ; done Produces output files with $i in the name: $i.eo68065. Is it possible to do what I want to do? Note: ideally, I would like to use basename $i rather than $i for the output file name.

I'd like my log files to be named after a variable. Since this isn't possible: #SBATCH --output some_software.${var}.out #SBATCH --error some_software.${var}.err I came across this work around but not sure if it's ok, or if it's wrong or if there's a better way: #!/bin/bash #SBATCH --job-name my_job #SBATCH --partition some_partition #SBATCH --nodes 1 #SBATCH --cpus-per-task 8 #SBATCH --mem 50G #SBATCH --time 6:00:00 #SBATCH --get-user-env=L #SBATCH --export=NONE var=$1 # SLURM doesn't support variables in #SBATCH --output, so handle logs manually if [ -z "$SLURM_JOB_ID" ]; then log_dir="/data/logs" sbatch --output="${log_dir}/some_software.${var}.out" \ --error="${log_dir}/some_software.${var}.err" \ "$0" "$var" exit fi /data/software/nextflow run /data/software/some_software/main.nf

I'd like to make a shell alias for customers to use like squeue --format="%.9i %10j %8u %8T %.12M %6D %20R %16P %.4C %m" The right-most %m format tag shows how much memory a job requested with the --mem=### argument, but squeue labels that output column as MIN_MEMORY. That is backwards, our IT has chosen to enforce memory policies that will OOM Kill anything that reached beyond its --mem= amount, so in this case that column should really say MAX_MEMORY. I have not found a different format tag to show it as an enforced maximum label, any chance there is a way to change that MIN_MEMORY label from %m to say something else for the label? It will be confusing for customer users to see MIN_MEMORY at the same time as they see OOM Killed... squeue --format="%.9i %10j %8u %8T %.12M %6D %20R %16P %.4C %m" CLUSTER: cluster-01 JOBID NAME USER STATE TIME NODES NODELIST(REASON) PARTITION CPUS **MIN_MEMORY**

I have a question for those of you familiar with the scheduler Slurm. Sometimes I get the following error message slurmstepd: error: Exceeded step memory limit at some point. I know it means the memory allocated to my process wasn't enough. Nonetheless, the process isn't killed by the scheduler and often times it seems innocuous: The program runs to completion and the output files look in good shape. Should I **always** assume that output is faulty and rerun the programs if I get that error message? Why sometimes the allocated memory can be exceeded but the program isn't killed?

I have an account on a compute cluster which uses the SLURM scheduler. I have some jobs in the queue and I'm using the "watch" command to see their status: watch squeue -u myUserName Does constantly running this command put any significant stress on the scheduler?

When I specify in gres.conf to omit the first GPU, Processes in Slurm still use the first one. If I allow Slurm to manage both, the second concurrent process properly goes onto the second GPU. Why? Context: I have a server with ubuntu 20.4.2 LTS with Slurm (23.11.4 from apt) installed. The server has 2 GPUS: Wed Apr 2 12:13:38 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4070 Off | 00000000:01:00.0 Off | N/A | | 30% 29C P8 6W / 200W | 10468MiB / 12282MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 4070 Off | 00000000:03:00.0 Off | N/A | | 30% 29C P8 3W / 200W | 158MiB / 12282MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ I intend to take out the first GPU out of local Slurm, but encountered an issue. According to the docs: https://slurm.schedmd.com/gres.conf.html , I should be able to just omit the GPU in gres.conf, eg from this: NodeName=server2 Name=gpu Type=geforce_rtx_4070 File=/dev/nvidia[0-1] To this: NodeName=server2 Name=gpu Type=geforce_rtx_4070 File=/dev/nvidia1 Since doing that alone causes the node to be DRAINED due to count mismatch I also modified slurm.conf compute node resources line from NodeName=server2 (...) Gres=gpu:geforce_rtx_4070:2 (...) to NodeName=server2 (...) Gres=gpu:geforce_rtx_4070:1 (...) The problem is that after those changes, all queued processed (by sbatch --gres=gpu:1 ...) go now onto the first GPU. I would assume that it's some problem with the process completely ignoring boundries made by Slurm, but with the old configuration, when launching two processes, the second launched process properly sees and uses only the second GPU. What might be the reason?

On a slurm cluster, is there ever a time when it’s **appropriate** to use sbatch *inside* an sbatch script? Or is it always a bad pattern? I’ve seen this in use, and it looks iffy:

#SBATCH -J RecursiveSbatchInvocation
things=(apples bananas carrots)
for thing in "${things[@]}"; do
  sbatch --wrap ./myscript.sh "$thing"
done

Compared to the two alternatives below, - Is there a danger to running sbatch inside sbatch? Will it potentially hog a lot of resources uselessly, or run very slowly for me, or be impolite to other users on the cluster, or have unpredictable/bad side effects? - Is there a realistic scenario where something *can only* be done with sbatch inside sbatch? **Alternative 1:** job array:

#SBATCH -J JobArrayInvocation
#SBATCH -a 0-2
things=(apples bananas carrots)
./myscript.sh "${things["$SLURM_ARRAY_TASK_ID"]}"

**Alternative 2:** srun inside sbatch with multiple tasks:

#SBATCH -J SrunInvocations
#SBATCH --ntasks=3
things=(apples bananas carrots)
for thing in "${things[@]}"; do
  srun --ntasks=1 ./myscript.sh "$thing" &
done

Somebody asked a similar question over on StackOverflow: “[**Can** I call sbatch recursively?](https://stackoverflow.com/questions/72747837/can-i-call-sbatch-recursively)” The two points in the discussion were (1) the cluster probably won’t allow it and (2) you can do the same thing with srun inside sbatch. But in my case the cluster _is_ allowing it. So I know it’s possible and I know “more canonical” ways to do this kind of thing, but I wonder if there’s a specific thing I can show people to say “please never use sbatch recursively.”

I am trying something odd. I have a slurm cluster configured that has 4 compute nodes. 2 of which are Windows 11 machines running WSL2 and I have it working. right now, I am trying to add GPU support to the SLURM cluster. For the 2 compute nodes that are native ubuntu 22.04 systems, the nvidia devices are showing as /dev/nvidia[0-3] but for the WSL2 (Ubuntu 22.04) there is nothing in /dev/ that speak to gpus. The hardware on the 2 PCs are different, but they are both running Windows 11 with Nvidia Driver 537.13, WSL2 Ubuntu 22.04, and cuda drivers installed as prescribed here: [NVidia Site](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=WSL-Ubuntu&target_version=2.0&target_type=deb_local) the output of nvidia-smi on both machines:

Thu Sep  7 23:28:30 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:01:00.0 Off |                  N/A |
|  0%   24C    P8              11W / 420W |     53MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  | 00000000:2E:00.0 Off |                  N/A |
|  0%   24C    P8              14W / 420W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        On  | 00000000:41:00.0  On |                  N/A |
|  0%   25C    P5              43W / 420W |   2753MiB / 24576MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        On  | 00000000:61:00.0 Off |                  N/A |
|  0%   25C    P8              12W / 420W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        23      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

Thu Sep  7 23:47:19 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:01:00.0 Off |                  N/A |
|  0%   25C    P8              29W / 260W |    433MiB / 11264MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

Both machines ls /dev/:

autofs           fuse          loop0   ptmx        ram5    stderr  tty19  tty32  tty46  tty6     vcs2   vcsu3
block            hugepages     loop1   ptp0        ram6    stdin   tty2   tty33  tty47  tty60    vcs3   vcsu4
bsg              hvc0          loop2   ptp_hyperv  ram7    stdout  tty20  tty34  tty48  tty61    vcs4   vcsu5
btrfs-control    hvc1          loop3   pts         ram8    tty     tty21  tty35  tty49  tty62    vcs5   vcsu6
bus              hvc2          loop4   ram0        ram9    tty0    tty22  tty36  tty5   tty63    vcs6   vfio
char             hvc3          loop5   ram1        random  tty1    tty23  tty37  tty50  tty7     vcsa   vhost-net
console          hvc4          loop6   ram10       rtc     tty10   tty24  tty38  tty51  tty8     vcsa1  virtio-ports
core             hvc5          loop7   ram11       rtc0    tty11   tty25  tty39  tty52  tty9     vcsa2  vport0p0
cpu_dma_latency  hvc6          mapper  ram12       sda     tty12   tty26  tty4   tty53  ttyS0    vcsa3  vport0p1
cuse             hvc7          mem     ram13       sdb     tty13   tty27  tty40  tty54  ttyS1    vcsa4  vsock
disk             initctl       mqueue  ram14       sdc     tty14   tty28  tty41  tty55  ttyS2    vcsa5  zero
dri              kmsg          net     ram15       sg0     tty15   tty29  tty42  tty56  ttyS3    vcsa6
dxg              kvm           null    ram2        sg1     tty16   tty3   tty43  tty57  urandom  vcsu
fd               log           nvram   ram3        sg2     tty17   tty30  tty44  tty58  vcs      vcsu1
full             loop-control  ppp     ram4        shm     tty18   tty31  tty45  tty59  vcs1     vcsu2

Ive tried reinstalling cuda on both windows, and cuda in wsl2 (as prescribed above) and googling the issue. My question is "where can I map the nvidia devices like I would in /dev/nvidia0 on traditional ubuntu installs? in gres.conf for SLURM"

I'm in the process of setting up a singe-node Slurm workstation machine and I believe I followed the process closely and everything is working just fine. *See* below: **sudo systemctl restart slurmdbd && sudo systemctl status slurmdbd**

● slurmdbd.service - Slurm DBD accounting daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:15:43 CET; 10ms ago
       Docs: man:slurmdbd(8)
   Main PID: 2597522 (slurmdbd)
      Tasks: 1
     Memory: 1.6M (peak: 1.8M)
        CPU: 5ms
     CGroup: /system.slice/slurmdbd.service
             └─2597522 /usr/sbin/slurmdbd -D -s

Mar 09 17:15:43 NeoPC-mat systemd: Started slurmdbd.service - Slurm DBD accounting daemon.
Mar 09 17:15:43 NeoPC-mat (slurmdbd): slurmdbd.service: Referenced but unset environment variable evaluates to an empty string: SLURMDBD_OPTIONS
Mar 09 17:15:43 NeoPC-mat slurmdbd: slurmdbd: Not running as root. Can't drop supplementary groups
Mar 09 17:15:43 NeoPC-mat slurmdbd: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version is: 5.5.5-10.11.8-MariaDB-0ubuntu0.24.04.1

**sudo systemctl restart slurmctld && sudo systemctl status slurmctld**

● slurmctld.service - Slurm controller daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:15:52 CET; 11ms ago
       Docs: man:slurmctld(8)
   Main PID: 2597573 (slurmctld)
      Tasks: 7
     Memory: 1.8M (peak: 2.8M)
        CPU: 4ms
     CGroup: /system.slice/slurmctld.service
             ├─2597573 /usr/sbin/slurmctld --systemd
             └─2597574 "slurmctld: slurmscriptd"

Mar 09 17:15:52 NeoPC-mat systemd: Starting slurmctld.service - Slurm controller daemon...
Mar 09 17:15:52 NeoPC-mat (lurmctld): slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS
Mar 09 17:15:52 NeoPC-mat slurmctld: slurmctld: slurmctld version 23.11.4 started on cluster mat_workstation
Mar 09 17:15:52 NeoPC-mat systemd: Started slurmctld.service - Slurm controller daemon.
Mar 09 17:15:52 NeoPC-mat slurmctld: slurmctld: accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd

**sudo systemctl restart slurmd && sudo systemctl status**

● slurmd.service - Slurm node daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:16:02 CET; 9ms ago
       Docs: man:slurmd(8)
   Main PID: 2597629 (slurmd)
      Tasks: 1
     Memory: 1.5M (peak: 1.9M)
        CPU: 13ms
     CGroup: /system.slice/slurmd.service
             └─2597629 /usr/sbin/slurmd --systemd

Mar 09 17:16:02 NeoPC-mat systemd: Starting slurmd.service - Slurm node daemon...
Mar 09 17:16:02 NeoPC-mat (slurmd): slurmd.service: Referenced but unset environment variable evaluates to an empty string: SLURMD_OPTIONS
Mar 09 17:16:02 NeoPC-mat slurmd: slurmd: slurmd version 23.11.4 started
Mar 09 17:16:02 NeoPC-mat slurmd: slurmd: slurmd started on Sun, 09 Mar 2025 17:16:02 +0100
Mar 09 17:16:02 NeoPC-mat slurmd: slurmd: CPUs=16 Boards=1 Sockets=1 Cores=8 Threads=2 Memory=128445 TmpDisk=575645 Uptime=2069190 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
Mar 09 17:16:02 NeoPC-mat systemd: Started slurmd.service - Slurm node daemon.

If needed, I can attach the results for the corresponding journalctl, but no error is shown other than these two messages slurmd.service: Referenced but unset environment variable evaluates to an empty string: SLURMD_OPTIONS and slurmdbd: Not running as root. Can't drop supplementary groups in the journalctl -fu slurmd and in the journalctl -fu slurmdbd, respectively. For some reason, however, I'm unable to run sinfo in a new tab even after setting the link the *slurm.conf* in my *.bashrc*... this is what I'm prompted with > sinfo: error: Couldn't find the specified plugin name for auth/munge > looking at all files sinfo: error: cannot find auth plugin for > auth/munge sinfo: error: cannot create auth context for auth/munge > sinfo: fatal: failed to initialize auth plugin which seems to depend on munge but I'm cannot really understand to what specifically — it is my first time installing Slurm. Any help is much appreciated, thanks in advance!

In a slurm system, when I use **srun** command to run the program. It runs very slow and seems like only one processor works. srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16 $FEAPHOME8_3/parfeap/feap -log_summary lu.log But if I write a **sbatch** script, it can run very quickly and looks like all the processors work. #!/bin/sh -l #SBATCH --job-name=test #SBATCH --account=free #SBATCH --nodes=1 #SBATCH --ntasks-per-node=24 #SBATCH --cpus-per-task=1 #SBATCH --exclusive #SBATCH --time=6:00:00 echo ' ' echo ' ****** START OF MAIN-JOB ******' date srun -n 16 echo y | mpirun -np 16 $FEAPHOME8_3/parfeap/feap -log_summary lu.log echo ' ****** END OF MAIN-JOB ******' #End of script Could anybody please tell me what's going on?

Hi I was working on installing Slurm and got most things sorted out but upon launching

journalctl -fu slurmdbd

I get the following: > Jan 25 12:49:49 ... systemd: Stopped slurmdbd.service - Slurm DBD > accounting daemon. Jan 25 12:49:49 ... systemd: Started > slurmdbd.service - Slurm DBD accounting daemon. Jan 25 12:49:49 ... > (slurmdbd): slurmdbd.service: Referenced but unset > environment variable evaluates to an empty string: SLURMDBD_OPTIONS > Jan 25 12:49:49 ... slurmdbd: slurmdbd: error: Unable to open > pidfile `/run/slurmdbd.pid': Permission denied Jan 25 12:49:49 ... > slurmdbd: slurmdbd: Not running as root. Can't drop > supplementary groups Jan 25 12:49:49 ... slurmdbd: slurmdbd: > accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server > version is: 8.0.40-0ubuntu0.24.04.1 Jan 25 12:49:49 ... > slurmdbd: slurmdbd: error: mysql_query failed: 1193 Unknown > system variable 'wsrep_on' Jan 25 12:49:49 ... slurmdbd: > select @@wsrep_on; Jan 25 12:49:49 ... slurmdbd: slurmdbd: > error: mysql_db_get_var_str: null result from query `select > @@wsrep_on;` Jan 25 12:49:49 ... slurmdbd: slurmdbd: slurmdbd > version 23.11.4 started which I assume being the cause for sinfo not displaying correctly with the following message: > sinfo: error: Couldn't find the specified plugin name for auth/munge > looking at all files sinfo: error: cannot find auth plugin for > auth/munge sinfo: error: cannot create auth context for auth/munge > sinfo: fatal: failed to initialize auth plugin To be noted that munge seems to be working correctly, see below, and therefore I cannot pinpoint the cause of Slum not showing sinfo option other than the issue I presented before. I also attach my **slurm.conf** and **slurmdbd.conf** for clarity in case helpful; this should be the recommended installation for a single node machine. Thanks in advance! **slurm.conf**

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=
SlurmctldHost=
MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=2
SlurmctldPidFile=/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/cgroup
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU_Memory
#
#ACCOUNTING STORAGE PORT
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=
AccountingStoreFlags=job_comment
JobCompType=jobcomp/none
JobAcctGatherType=jobacct_gather/cgroup
JobAcctGatherFrequency=30
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
#
# COMPUTE NODES
NodeName= CoresPerSocket=8 ThreadsPerCore=2 RealMemory=126397 Sockets=1
PartitionName=local Nodes=ALL Default=YES MaxTime=INFINITE State=UP

**slurmdbd.conf**

=auth/munge
DbdHost=localhost
DebugLevel=info
StorageHost=localhost
StorageLoc=slurm_acct_db
StoragePass=slurmdbpass
StorageType=accounting_storage/mysql
StorageUser=slurm
LogFile=/var/log/slurm/slurmdbd.log
PidFile=/run/slurmdbd.pid
SlurmUser=slurm

P.S. output of munge -n | unmunge

STATUS:          Success (0)
ENCODE_HOST:      (127.0.1.1)
ENCODE_TIME:     2025-02-11 09:27:00 +0100 (1739262420)
DECODE_TIME:     2025-02-11 09:27:00 +0100 (1739262420)
TTL:             300
CIPHER:          aes128 (4)
MAC:             sha256 (5)
ZIP:             none (0)
UID:              (1000)
GID:              (1000)
LENGTH:          0

I am working on understanding of how cgroups memory resource controller is enabled on Ubuntu 20.04. I have several Ubuntu machines that make up a Slurm 23.02.7 cluster. In cgroup.conf , SchedMD states : > Debian and derivatives (e.g. Ubuntu) usually exclude the memory and memsw (swap) cgroups by default. To include them, add the following parameters to the kernel command line: cgroup_enable=memory swapaccount=1 If I follow this advise and do :

root@node01:~# grep CONFIG_MEMCG /boot/config-5.4.0-169-generic
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
# CONFIG_MEMCG_SWAP_ENABLED is not set
CONFIG_MEMCG_KMEM=y

It seems that contrary to SchedMD's opinion, the cgroups memory controller is already enabled by default. However, I'd like formal documentation about what exactly every variable is in /boot/config-5.4.0-169-generic. I've searched for an associated man page, but cannot find one. **Question** : 1. What man page or other documentation exists describing the configurations set in /boot/config-5.4.0-169-generic?

I'm getting this error "sbatch: error: Batch job submission failed: Invalid node name specified" for this bash script #!/bin/bash -l #SBATCH --gpus=1 #SBATCH -p overcap #SBATCH -A overcap #SBATCH --signal=USR1@120 #SBATCH --time=10:00 #SBATCH --requeue #SBATCH --nodelist=brainiac,omgwth,cyborg,sonny,robby,spd-13,qt-1,dave,nestor,crushinator,deebot,xaea-12,baymax,megabot,randotron,chappie,heistotron,roberto,herbie,shakey,chitti,samantha,clippy,kitt,tachikoma #SBATCH -o err_test.out srun python src/train.py is there a way to find out which node is invalid in the list?

I'm working with a job array where I'm controlling for the potential execution of the various steps of a script multiple times. In this case, only the missing ones will be processed using appropriate if statements. As part of the script I need to rename folders before the last command, so I wish for the if statements to check also the newly named folders on subsequent runs; however, I'm missing the logic on how to do so... Below, an example of the folders I visit with the array the first time

/path/to/INLUP_00165
/path/to/INLUP_00169
/path/to/INLUP_00208
/path/to/INLUP_00214
/path/to/INLUP_00228
/path/to/INLUP_00245
/path/to/INLUP_00393
/path/to/INLUP_00418

which will became, after the first execution, the following

/path/to/INLUP_00165-35
/path/to/INLUP_00169-27
/path/to/INLUP_00208-35
/path/to/INLUP_00214-32
/path/to/INLUP_00228-32
/path/to/INLUP_00245-34
/path/to/INLUP_00393-29
/path/to/INLUP_00418-32

This is the script I'm using

#!/bin/bash
#SBATCH --nodes=1 --ntasks=1 --cpus-per-task=12
#SBATCH --time=200:00:00
#SBATCH --mem=80gb
#
#SBATCH --job-name=name1
#SBATCH --output=name_1.out
#SBATCH --array=[1-8]%8
#
#SBATCH --partition=bigmem
#SBATCH --exclude=node5

NAMES=$1
h=$(sed -n "$SLURM_ARRAY_TASK_ID"p $NAMES)

#load modules

ID="$(echo ${h}/*.fastq.gz | cut -f 1 -d '.' | sed 's#/path/to/INLUP_[0-9]\+/##')"
readarray -t cond  $h/log.txt; find . -maxdepth 1 -type f,l -not -name '*.filt.fastq.gz' -not -name '*.txt' -not -name '*.bin' -delete

		HIGH=$(grep '\[M::ha_analyze_count\] highest\:' $h/log.txt | tail -1 | sed 's#\[M::ha_analyze_count\] highest\: count\[##' | sed 's#\] = [0-9]\+$##') #highest peak
		LOW=$(grep '\[M::ha_analyze_count\] lowest\:' $h/log.txt | tail -1 | sed 's#\[M::ha_analyze_count\] lowest\: count\[##' | sed 's#\] = [0-9]\+$##') #lowest peak
		SUFFIX="$(echo $(( ($HIGH - $LOW) "*" 3/4 + $LOW )))" #estimated homozygous coverage

		mv $h $h-${SUFFIX}

		HOM_COV="$(echo ${h} | sed 's#/path/to/INLUP_[0-9]\+-##')"

		last command tool &> $h-${SUFFIX}/log_param.txt
fi

I thought to attempt, with the next code block, to parse the number after the - in the new folders' name by storing it into an array to check element by element — starting with the first and assigning it to the first folder and so on.

readarray -t cond < <(
    for filename in INLUP_00*
    do 
        printf "$filename \n" | sed 's#INLUP_[0-9]\+-##'
    done
)

How can I link it to the file I feed as input which is unchanged and contains the original paths to the folders? Maybe something related to the way a path is associated to the TASK ID like here h=$(sed -n "$SLURM_ARRAY_TASK_ID"p $NAMES). Let me know, thanks in advance!

I keep overthinking about how can I optimize my pipeline. Essentially, I have multiple tools that will be executed on two haplotypes (**hap1** and **hap2**) for a plant species. The general structure is as follows:

> tree INLUP_00001
INLUP_00001
├── 1.pre_scaff
├── 2.post_scaff
├── INLUP00233.fastq.gz
├── INLUP00233.filt.fastq.gz
├── INLUP00233.meryl
├── hap1
├── hap2
└── hi-c
    ├── INLUP00001_1.fq.gz
    └── INLUP00001_2.fq.gz

(I will have 16 of these INLUP_????? parent directories) So, with this in mind I organized a job array which reads from the following file

path/to/INLUP_00001/hap1
path/to/INLUP_00001/hap2
path/to/INLUP_00002/hap1
path/to/INLUP_00002/hap2
.
.
.

where I have a variable – ${HAP} – that discriminates which haplotype I'm working on, in which sub-directory the data will be written, and eventual names for each output. This seems to best optimize runtime and resource allocation. However, there is a problem with the very first tool I'm using; this application is the one generating both **hap1** and **hap2** and does not accept the ${HAP} variable. In other words, I have no control on the outputs based on my job array list which will redundantly execute this single command 32 times not only causing issues but also wasting time and resources... Is there a way to control for the execution of this command only one time for each INLUP sample while preserving the control on haplotypes with the ${HAP} variable within the job array? I thought about alternatives with for cycles applied to all other tools in the pipeline to accommodate **hap1** and **hap2**, but they ended up making the script overly long in my opinion and more complex... also the resources allocated for the first tool cannot be easily partitioned/assigned to independent tasks for **hap1** and **hap2** for the other tools. Any idea/help is much appreciated, sorry for the long message if more context is needed I can provide a short MWE of the first few commands.

I am new to Slurm. I have set it up in the cluster and on some nodes of a partition, the job runs perfectly fine but some other nodes of the same partition, the jobs do not run. They get cancelled the moment I submit them. When I ssh into the node on which the jobs fail and look at the slurmd.log I see:

[2024-08-30T01:49:30.986] [43095.batch] error: Could not open stdout file /data/vmahajan/larchtest/slurm-43095.out: No such file or directory
[2024-08-30T01:49:30.986] [43095.batch] error: _fork_all_tasks: IO setup failed: Slurmd could not connect IO
[2024-08-30T01:49:30.987] [43095.batch] get_exit_code task 0 died by signal: 53
[2024-08-30T01:49:30.989] [43095.batch] done with job

I can only conclude that Slurm is not able to write the out file, but I fail to understand why so only on some nodes it fails to do so.

When running a SLURM job using sbatch, slurm produces a standard output file which looks like slurm-102432.out (slurm-jobid.out). I would like to customise this to (yyyymmddhhmmss-jobid-jobname.txt). How do I go about doing this? Or more generally, how do I include computed variables in the sbatch argument -o? I have tried the following in my script.sh #SBATCH -p core #SBATCH -n 6 #SBATCH -t 1:00:00 #SBATCH -J indexing #SBATCH -o "/home/user/slurm/$(date +%Y%m%d%H%M%S)-$(SLURM_JOB_ID)-indexing.txt" but that did not work. The location of the file was correct in the new directory but the filename was just literal line $(date +%Y%m%d%H%M%S)-$(SLURM_JOB_ID)-indexing.txt. So, I am looking for a way to save the standard output file in a directory /home/user/slurm/ with a filename like so: 20160526093322-10453-indexing.txt