slurm | Slurm: A Highly Scalable Workload Manager | Continuous Deployment library

I am playing with a cluster using SLURM on AWS. I have defined the following parameters :

I need to submit a slurm array that will run the same script 18000 times (for independent genes), and I wanted to do this in a way that won't cause problems for my Uni's cluster.

Currently, the MaxArraySize set by the admins is 2048. I was going to manually set my options like:

First array script:

What is the proper way of configuring jupyter on a server with slurm? After reading the docs, I am excecuting my python script through slurm like this (I am not sure if this is valid):

please would you help me with your suggestions on the following :

<> I am using an account on a SLURM cluster where the storage space of my home directory (ie. /home/user) is maximum 32 GB

<> I am running on the SLURM cluster a singularity container that is working only if the the input files are located in the home directory (/home/user that has 32 GB), instead of using any other location on the SLURM cluster (i.e. /labs/professor where we have 7TB)

The question is : is there any way to set up the large account space (i.e. /labs/professor) as a "home directory" in order for the singularity container to run ?

thanks a lot, with much appreciation,

-- bogdan

I am new to HPC and SLURM especially, and i ran into some troubles.

I was provided with acces to a HPC cluster with 32 CPUs on each node. In order to do the needed calculations I made 12 Python multiprocessing Scripts, where each Script uses excactly 32 CPU's. How, instead of starting each Script manually in the interactive modus ( which is also an option btw. but it takes a lot of time) I decided to write a Batch Script in order to start all my 12 Scripts automatically.

//SCRIPT//

#!/bin/bash

#SBATCH --job-name=job_name

#SBATCH --partition=partition

#SBATCH --nodes=1

#SBATCH --time=47:59:59

#SBATCH --export=NONE

#SBATCH --array=1-12

module switch env env/system-gcc module load python/3.8.5

source /home/user/env/bin/activate

python3.8 $HOME/Script_directory/Script$SLURM_ARRAY_TASK_ID.py

exit

//UNSCRIPT//

But as far as i understand, this script would start all of the Jobs from the Array on the same node and thus the underlying python scripts might start a "fight" for the available CPU's and thus slow down.

How should i modify my bash file in Order to start each task from the array on a separate node?

Thanks in advance!

So I am running a slurm job across 2 laptops (2 nodes), and I notice both laptop gets insanely laggy, to the point that the mouse could not even move. When I uses htop I saw the job was using 4 cores in 100% usage. I know for sure that the job does not take up that many cpu usage. How do I make slurm so that it uses only the required amount of cpu power?

I have an aws HPC auto scale cluster managed by slurm, I can submit jobs using sbatch, however I want to use spraklyr on this cluster so that slurm increases the cluster size based on the workload of the sparklyr code in the R script. Is this possible?

is it possible to use variable expansion in #SBATCH lines in slurm? for instance I want to have line below:

I'm new to SLURM and I'm trying to do something very natural: I have a compiled C program, off.exe, which takes one variable as input, and I want to run it several times in parallel, each with a different value of the input parameter. I thought I could use the %a array iterator as input: