qstat | New official qstat repository | Game Engine library

The title kind of says it all. I'm looking for a command line test to check if either SLURM, or MOAB/Torque is available for submitting jobs too.

My thought is to check if the command qstat finishes with exit code, or if squeue finished with exit code zero. Would this be the best way of doing this?

I have a rather complex python algorithm I need to distribute across a HPC cluster.

The code is run from a Jupyterhub instance with 60 gb memory. The configuration of the PBS cluster is 1 process, 1 core, 30Gb per worker, nanny=False (the computations won't run otherwise) for a total of 26 workers (the total memory is about 726GB)

I do not need to fetch back any data, since what is needed is written to disk right at the end of the computations. Note that each computations takes about 7 minutes when run independantly.

The problem I run into is the following : each independant worker (Jobname : dask-worker) seems to run fine, it has about 20Gb available of which max 5Gb is used. But whenever I try to launch more than about 50 jobs, then the central worker (Jobname : jupyterhub) runs out of memory after about 20 minutes.

Here is how I distribute the computations :

Does anyone have a simple text list or regex of all default JavaScript commands, functions, build-ins and/or keywords?

I am writing an AI to detect malicious web calls. To make it easier for the AI to learn I am pre detecting SQL, Unix etc.

What I have:

We have switched to using slurm from sge for our cluster job queuing system. In sge when you used the qstat function it printed the number of cpus/slots in use per job - is there a simple way to do this in slurm? squeue only shows the number of nodes used per job. Thanks.

I want to use dpdk pktgen to read a pcap file with IP packets. Then replay each IP packet 10 times with unique IP address. Example if my pcap file has 20 packets, starting with TCP SYN packet, then I want to replay each packet with source IPs in my list, thus total of 10 x 20 packets should be sent. Basically, I want to resend buffers stored in rte_mbuf again and again, after modifying source IP in each run.

Please let me know what's the best way of doing this.

I first thought of editing the source IP in the rte_mbuf pointed by pkts before calling the rte_eth_tx_burst(info->pid, qid, pkts, cnt) function by having a source IP incrementing loop but it seems that rte_eth_tx_burst function frees up the rte_mbuf pointed by pkts after sending the packet once so I cannot edit the pkts buffer again.

Why is it not recommended to run squeue in a loop to avoid overloading Slurm, but no such limitations are mentioned for the bjobs tool from LSF or qstat from SGE ?

The man page for squeue states:

PERFORMANCE

Executing squeue sends a remote procedure call to slurmctld. If enough calls from squeue or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service.

Do not run squeue or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

which to my understanding disapproves the use of e.g. watch squeue. Such a warning is commonly found in site-specific documentation, e.g. here:

Although squeue is a convenient command to query the status of jobs and queues, please be careful not to issue the command excessively, for example, invoking the query for the status of a job every five seconds or so using a script after a job is submitted.

In comparison, I could find no such warning for similar tools on other engines e.g. qstat or bjobs. I see people using all of these tools in a repetitive fashion without distinction, e.g. here for squeue, here for bjobs.

The quote above from Slurm documentation mention a RPC, is it a way of doing different from other engines ? Is there an architecture difference between Slurm and other grid engines that makes querying the status of all jobs more costly ?

I'm trying to write a script that builds a list of nodes then ssh into the first node of that list and runs a checknodes.sh script which it's self is just a for i loop that calls checknode.sh

The first 2 lines seems to work ok, the list builds successfully, but then I get either get just the echo line of checknodes.sh to print out or an error saying cat: gpcnodes.txt: No such file or directory

MYSCRIPT.sh: