mpirun | MPIRUN wrapper script to generate and execute an MPIRUN

In Slurm the number of tasks is essentially the number of parallel programs you can start in your allocation. By default, each task can access one CPU (which can be core or thread, depending on config), which can be modified with --cpus-per-task=#.

This in itself does not tell you anything about the number of nodes you will get. If you just specify --ntasks (or just -n), your job will be spread over many nodes, depending on whats available. You can limit this with --nodes #min-#max/--nodes #exact. Another way to specify the number of tasks is --ntasks-per-node, which does exactly what is says and is best used in conjunction with --nodes. (not with --ntasks, otherwise it's the max number of tasks per node!)

So, if you want three nodes with 72 tasks (each with the one default CPU), try:

By the following setting, the above issue has been solved.

I don't view this as a big enough set of data to require mpi provided you take an efficient approach to processing the data.

As I mentioned in the comments, I find the best approach to processing large amounts of numerical data is first to use numpy vectorization, then try using numba jit compiling, then use multi-core processing as a last resort. In general that's following the order of easiest to hardest, and will also get you the most speed for the least work. In your case I think vectorization is truly the way to go, and while I was at it, I did some re-organization which isn't really necessary, but helped me to keep track of the data.

MPI_Bcast

Broadcasts a message from the process with rank "root" to all other processes of the communicator

is a collective communication routine, hence it should be called by all the processes in a given communicator. Therefore, you need to remove the following condition if(myrank%3==0) and then you need to adapt the root process accordingly, instead of using localRank.

In your current code, only the processes with myrank 0 and 3 called the MPI_Bcast (both belonging to different communicators). So process 0 calls

The problem has nothing to do with MPI and also nothing to do with the difference in cluster.¹ It is problematic code, that fails with gfortran but works under ifort by pure luck.

If the file is opened with a fixed record length (recl=...) a write statement must not exceed this length, even if the output goes to /dev/null. The fix is simply to not open with a fixed record length and omit the recl=... argument.

Apparently the runtime library of ifort is more permissive and even works if the byte length of the written object is larger than the record length specified in the open statement.

In the following example the last write statement fails under gfortran.

I hate to say so, but that answer by that sehe guy was just flawed. Thanks for finding it.

The problem was that it didn't store the number of non-zero cells during serialization. Oops. I don't know how I overlooked this when testing.

(Looks like I had several versions and must have patched together a Frankenversion of it that wasn't actually properly tested).

I also threw in a test the matrix is cleared (so that if you deserialize into an instance that had the right shape but wasn't empty you don't end up with a mix of old and new data.)

The proposed program is currently broken when using Open MPI, see issue https://github.com/open-mpi/ompi/issues/8763. The current workaround is to use MPICH.

Inside your loop, add the following code to the beginning of the loop body:

Which gives only about 25% performance gain. My guess here is that the bottleneck may be caused by processes that compete to access the memory. (..)

Your code is mostly communication- and CPU- bound. Moreover, according to your results for 2, 5, and 10 processes: