pymp | Easy , OpenMP style multiprocessing for Python on Unix

On Linux, a child process is created by fork method by default. That means, the child process inherits almost everything from the parent process.

On Windows, the child process is created by spawn method. That means, a child process is started almost from crash, re-imports and re-executes any code that is outside of the guard cloud if __name__ == '__main__'.

On Linux, because the logger object is inherited, your program will start logging. But it is far from perfect since you log directly to the file. Sooner or later, log lines will be overlapped or IO error on file happens due to race condition between processes.

On Windows, since you didn't pass the logger object to the child process, and it re-imports your pymp_global module, logger is a None object. So when you try logging with a None object, it crashes for sure.

Logging with multiprocessing is not an easy task. For it to work on Windows, you must either pass a logger object to child processes and/or log with QueueHandler. Another similar solution for inter-process communication is to use SocketHandler.

The idea is that only one thread or process does the logging. Other processes just send the log records. This prevents the race condition and ensures the log is written out after the critical process got time to do its job.

So how to implement it?
I have encountered this logging problem before and already written the code.
You can just use it with logger-tt package.