Pathos | File management and path analysis for Swift

I'm the pathos author. First off, you are using a ParallelPool, which uses ppft... which uses dill.source to convert objects to source code, and then passes the source code to the new process to then build a new object and execute. You may want to try a ProcessPool, which uses multiprocess, which uses dill, which uses a more standard serialization of objects (like pickle). Also, when you are serializing code (either with dill or dill.source) you should take care to make sure the code is as self-encapsulated as possible. What I mean is that:

You've got the wrong side of the stick. Take a look at multiprocessing module documents. As it says, calling Pool method is for running multiple instance of same function simultaneously (in parallel). So call Pool method as many numbers you want, meanwhile your method does not any parameters, call it without any arguments:

You are probably confused with the concept. The if __name__ == "__main__" guard in Python exists exactly in order for it to be possible for all Python files to be importable.

Without the guard, a file, once imported, would have the same behavior as if it were the "root" program - and it would require a lot of boyler plate and inter-process comunication (like writting a "PID" file at a fixed filesystem location) to coordinate imports of the same code, including for multiprocessing.

Just leave under the guard whatever code needs to run for the root process. Everything else you move into functions that you can call from the importing code.

If you'd run "all" the script, even the part setting up the multiprocessing workers would run, and any simple job would create more workers exponentially until all machine resources were taken (i.e.: it would crash hard and fast, potentially taking the machine to an unresponsive state).

So, this is a good pattern - th "dothejob" function can call all other functions you need, so you just need to import and call it, either from a master process, or from any other project importing your file as a Python module.

When using multiprocessing, the library spawns multiple different processes. Each process has its own address space. This means that each of those processes has its own copy of the variable, and any change in one process will not reflect in other processes.

In order to use shared memory, you need special constructs to define your global variables. For pathos.multiprocessing, from this comment, it seems you can declare multiprocessing type shared variables by simply importing the following:

Each "parallel" insertion process needs its own connector and cursor. You can't share them across any sort of thread.

You can use connection pooling to make faster the allocation and release of connections.

There's no magic in MySQL (or any DBMS costing less than the GDP of a small country) that lets it scale up to handle large scale data insertion on ~100 connections simultaneously. Paradoxically, more connections can have lower throughput than fewer connections, because of contention between them. You may want to rethink your system architecture so you can make it work OK with a few connections.

In other words: fewer bigger tables perform much better than many small tables.

Finally, read about ways of speeding up bulk inserts. For example this sort of multirow insert

I'm the author of pathos. Whenever you see self._value in the error, what's generally happening is that something you tried to send to another processor failed to serialize. The error and traceback is a bit obtuse, admittedly. However, what you can do is check the serialization with dill, and determine if you need to use one of the serialization variants (like dill.settings['trace'] = True), or whether you need to restructure your code slightly to better accommodate serialization. If the class you are working with is something you can edit, then an easy thing to do is to add a __reduce__ method, or similar, to aid serialization.

Instead of from pathos.multiprocessing import ProcessPool as Pool, I used from multiprocess import Pool, which is essentially the same thing. Then I tried some alternative approaches.

So:

So I have resolved this issue. I would still be great if someone like mmckerns or someone else with more knowledge than me on multiprocessing could comment on why this is a solution.

The issue seemed to have been that the Manager().list() was declared in __init__. The following code works without any issues:

Boost is not installed. You can try this