PathOS | clinical application for filtering | Genomics library

When using multiprocessing, the library spawns multiple different processes. Each process has its own address space. This means that each of those processes has its own copy of the variable, and any change in one process will not reflect in other processes.

In order to use shared memory, you need special constructs to define your global variables. For pathos.multiprocessing, from this comment, it seems you can declare multiprocessing type shared variables by simply importing the following:

Each "parallel" insertion process needs its own connector and cursor. You can't share them across any sort of thread.

You can use connection pooling to make faster the allocation and release of connections.

There's no magic in MySQL (or any DBMS costing less than the GDP of a small country) that lets it scale up to handle large scale data insertion on ~100 connections simultaneously. Paradoxically, more connections can have lower throughput than fewer connections, because of contention between them. You may want to rethink your system architecture so you can make it work OK with a few connections.

In other words: fewer bigger tables perform much better than many small tables.

Finally, read about ways of speeding up bulk inserts. For example this sort of multirow insert

I'm the author of pathos. Whenever you see self._value in the error, what's generally happening is that something you tried to send to another processor failed to serialize. The error and traceback is a bit obtuse, admittedly. However, what you can do is check the serialization with dill, and determine if you need to use one of the serialization variants (like dill.settings['trace'] = True), or whether you need to restructure your code slightly to better accommodate serialization. If the class you are working with is something you can edit, then an easy thing to do is to add a __reduce__ method, or similar, to aid serialization.

Instead of from pathos.multiprocessing import ProcessPool as Pool, I used from multiprocess import Pool, which is essentially the same thing. Then I tried some alternative approaches.

So:

So I have resolved this issue. I would still be great if someone like mmckerns or someone else with more knowledge than me on multiprocessing could comment on why this is a solution.

The issue seemed to have been that the Manager().list() was declared in __init__. The following code works without any issues:

Boost is not installed. You can try this

I'll try to attack the main issues. My understanding of your actual problem is quite limited as I don't know what your code actually does.

This doesn't work as pygame is built on SDL2 which states "You should not expect to be able to create a window, render, or receive events on any thread other than the main one."

Same as above, the display only work in the main thread.

You could pickle the methods using something like dill, but it fells (to me) wrong to copy full on python object between processes. I'd go for another solution.

You need to use to serialize Python objects in order to send them between processes.

I don't understand what this mean.

It seems to me that the problem could be solved with better separation of data and visualization. The training should have no knowledge about any visualization, as it's not dependent on how you want to display it. So there should not be any reason to share the pygame display.

Once this is done, it shouldn't be too much problem (multi-threading always causes problems) to do what you're trying to do. Regarding the pickle issue; I'd try to avoid pickling Python objects and functions, and instead just pass basic primitives between threads and processes. It seems like you should be able to assign self.fitnessFromArray with a simple int instead and based on its value do the min/avg/max calculation in the thread/process instead.

If you want to do threading, then the main thread will take care of the rendering. It'll also spawn threads for the training. When the threads are completed they'll return their result (or put it in a thread safe storage) and the main thread will poll the data and display the result. If the work done by the training takes longer than one frame, then divide up the work so each thread only partially train and can continue where it left off the next frame.

The principal is the same if you instead want separate processes. The main process starts up several training processes and connect to them via sockets. From the sockets, you'd poll information about the state of the program and display it. It would basically be a client-server architecture (albeit on localhost) where the training scripts are servers and the main process is a client.

Converting both list and dict variables to pathos.helpers.mp.Array without an intermediate pa.helpers.mp.Manager as suggested by @Mike McKerns brought the desired performance boost.

pathos leverages multiprocess, which has the same interface as multiprocessing, but uses dill. You can access it either of these ways.