multiprocess | common PHP/Python/js ... script change daemon

Pythons multiprocessing package is limited to shared memory parallelization. It spawns new processes that all have access to the main memory of a single machine.

You cannot simply scale out such a software onto multiple nodes. As the different machines do not have a shared memory that they can access.

To run your program on multiple nodes at once, you should have a look into MPI (Message Passing Interface). There is also a python package for that.

Depending on your task, it may also be suitable to run the program 4 times (so one job per node) and have it work on a subset of the data. It is often the simpler approach, but not always possible.

You have one issue with the given code. The blow line:

https://docs.python.org/3/library/math.html

The math module consists mostly of thin wrappers around the platform C math library functions.

While python itself can only execute a single instruction at a time, a low level c function that is called by python does not have this limitation.
So it's not python that is using multiple cores but your system's well optimized math library that is wrapped by python's math module.

That basically answers both your questions.

Regarding the usefulness of multiprocessing: It is still useful for those cases, where you're trying to parallelize pure python code or code that does not call libraries that already use multiple cores. However, it comes with inter process communication (IPC) overhead that may or may not be larger than the performance gain that you get from using multiple cores. Tuning IPC is therefore often crucial for multiprocessing in python.

While I do not know how to deal with this problem directly, I had a somewhat similar issue(and solved). The difference is:

• I use fairseq
• I can run my code on google colab with 1 GPU
• Got RuntimeError: unable to mmap 280 bytes from file : Cannot allocate memory (12) immediately when I tried to run it on multiple GPUs.

From the other people's code, I found that he uses python -m torch.distributed.launch -- ... to run fairseq-train, and I added it to my bash script and the RuntimeError is gone and training is going.

So I guess if you can run with 21000 samples, you may use torch.distributed to make whole data into small batches and distribute them to several workers.

This is not necessarily the best way to do it, but the shortest in writitng time. I don't know if you are familiar with python generators, but we will have to use one. the generator will work as a work dispatcher.

You didn't specify what platform you are running under (you are supposed to tag your question with your platform whenever you tag a question with multiprocessing), but it appears you are running under a platform that uses spawn to create new processes (such as Windows). This means that when a new process is launched, an empty address space is created, a new Python interpreter is launched and the source is re-executed from the top.

So although you have in the block that begins if __name__ == '__main__': assigned to shared_list a managed list, each process in the pool that is created will be executing shared_list = [] clobbering your initial assignment.

You can pass shared_list as the first argument to your worker function:

Change the format of CountryList:

You should create some kind of structure where you store the needed variables and process handles. Block with join after that loop until all subprocesses are finished and then work with the resulted files.

Child processes in Python do not share memory by default—code running in one process cannot access or change the memory used by another process. This means that each process has its own copy of each of the variables you are using, including the mp_worker_class.nbiter variable. Hence, you don't see changes that a child process makes to its mp_worker_class.nbiter variable from the parent process (or any of the other child processes, for that matter.)

As you have seen, we can get data from a parent process to a child process by using the args keyword argument to the multiprocessing.Process constructor. However, this simply copies the data from the parent to the child; we're still not sharing memory between the two processes.