aws-mpi-benchmark | MPI Benchmark on AWS HPC cluster

For most MPIs except Intel-MPI, we use Spack environment to simplify the installation and environment management. spack env activate discovers the MPI executable path, functionally similar to module load. Following sections install the environments openmpi3, openmpi4, mpich3. They will be shown in spack env list. Add pre-installed Slurm to Spack by copying scripts/spack_config/packages.yaml to ~/.spack/packages.yaml. The above installation uses TCP sockets. Libfabric & EFA can be enabled via spack install openmpi+pmi schedulers=slurm fabrics=libfabric ^libfabric fabrics=efa, but OpenMPI + EFA is currently quite slow for collectives due to this issue. Update: The problem is solved by libfabric 1.9 https://github.com/aws/aws-parallelcluster/issues/1143#issuecomment-561919602, which is added to Spack by https://github.com/spack/spack/pull/13964. We use the pre-installed Intel-MPI on AWS ParallelCluster, to avoid the hassle of configuring Intel-MPI with libfabric and EFA.
Launch by srun. OpenMPI needs to be compiled with --with-pmi (already handled by Spack).
Launch by orterun inside slurm interactive session or sbatch script. Still need to compile with --with-slurm.
Launch by srun. Need to set export I_MPI_PMI_LIBRARY=/opt/slurm/lib/libpmi.so
Launch by mpirun inside slurm interactive session or sbatch script. Should unset I_MPI_PMI_LIBRARY
https://slurm.schedmd.com/mpi_guide.html
https://www.open-mpi.org/faq/?category=slurm
https://software.intel.com/en-us/mpi-developer-guide-linux-job-schedulers-support