NYU-HPC | Handy scripts to run Spark jobs

NYU-HPC is a Shell library. NYU-HPC has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Handy scripts and other useful tricks to run jobs on high performance computing Greene cluster at New York University (but should be applicable elsewhere). Singularity is similar to Docker container, allows OS level virtualization. To see example of using Singularity, installing conda env in a container, submit the job with slurm, see singularity.md. If you are running a job on slurm, the output will be saved to a file e.g jobid.out but you don't see any stdout until the job is done. To fix this, modify print statement in python from print(something) to print(something,flush=True) reference: see a comment in Squash file system allows compressing a folder containing a large number of file into a .sqf file, which then can be mounted like a partition (think of a DVD). This is useful for decreasing inode number, and data is read only. myfolder is now avaiable as a mounted volume at /myfolder. For squashing a lot of folders, step 1 could be tedious so I wrote a script to loop through folders squashing.sh.