starmap

You should stop trying to invent the wheel, and instead start to leverage the built-in capabilities of Azure Databricks. Because Apache Spark (and Databricks) is the distributed system, machine learning on it should be also distributed. There are two approaches to that:

1. Training algorithm is implemented in the distributed fashion - there is a number of such algorithms packaged into Apache Spark and included into Databricks Runtimes

2. Use machine learning implementations designed to run on a single node, but train multiple models in parallel - that what typically happens during hyper-parameters optimization. And what is you're trying to do

Databricks runtime for machine learning includes the Hyperopt library that is designed for the efficient finding of best hyper-parameters without trying all combinations of the parameters, that allows to find them faster. It also include the SparkTrials API that is designed to parallelize computations for single-machine ML models such as scikit-learn. Documentation includes a number of examples of using that library with single-node ML algorithms, that you can use as a base for your work - for example, here is an example for scikit-learn.

P.S. When you're running the code with multiprocessing, then the code is executed only on the driver node, and the rest of the cluster isn't utilized at all.

I wouldn't call it exactly criminal, but I would call it malpractice.

Your code in thread_function constitutes a critical section whose execution needs to be serialized so that only a single process can be executing it at a time. Even what appears to be a single statement, shared_resource[val] += 1, consists of multiple bytecode instructions and two processes could read the same initial value of shared_resource[val] and store the same updated value. But clearly multiple processed running in parallel could clearly find that there are no keys in the dictionary and will be storing identical keys.

try running this changing with Pool(2) as p with:

with Pool(1) as p , with Pool(2) as p , with Pool(4) as p , with Pool(8) as p

and have a look at the different outputs

Yes. Pool.starmap() - and Pool.map() - return a result list with results in left-to-right order of the function applied to the iterable of arguments. Same way, e.g., the built-in map(), and itertools.starmap(), work in this respect, although those return generators rather than lists.

The only function of this kind that does not guarantee result order is Pool.imap_unordered(), which returns an iterable which defines nothing about the order in which it returns results.

If I'm understanding your implementations correctly, you haven't actually partitioned the arrays in any of these cases. so all you've done is run the exact same workflow, but on a different thread, so the "parallel" execution time is the original runtime plus the overhead of setting up the distributed job scheduler and passing everything to the second thread.

If you want to see any total time improvements, you'll have to actually rewrite your code to operate on subsets of the data.

In the dask case, use dask.array.from_numpy to split the array into multiple chunks, then rewrite your workflow to use dask.array operations rather than numpy ones. Alternatively, partition the data yourself and run your function on subsets of the array using dask distributed's client.map (see the quickstart).

None of these approaches will be easy, and you need to recognize that there is an overhead (both in terms of actual compute/network usagee/memory etc as well as a real investment of your time) in any of these, but if total runtime is important then it'll be worth it. See the dask best practices documentation for more background.

Update:

After your iteration with dask.array, your implementation is now faster than the single-threaded wall time, and yes, the additional CPU time is overhead. For your first time trying this, getting it to be faster than numpy/scipy (which, by the way, are already heavily optimized and are likely parallelizing under the hood in the single-threaded approach) is a huge win, so pat yourself on the back. Getting this to be even faster is a legitimate challenge that is far outside the scope of this question. Welcome to parallelism!

Additional reading:

• Dask best practices
• Understanding dask performance

This answer contains all the information you need: To retrieve results while they are being generated, imap_unordered is probably the best function to use as it returns results to the main thread once they are completed. You would just have to perform a bit of bookkeeping to ensure that the results end up in the right position in your result queue. A way to achieve that would be to pass an index to the parallelized function which that function then returns.

Some simplified pseudo-code below that you should be able to derive a solution with:

This doesn't answer your question but I'd be curious to know if it speeds things up. Just returning dictionaries instead of DataFrames should be much more efficient:

This happens because the worker processes attempt to import sumP, which also creates another 3 processes, ad-infinitum.

You'll need to put a guard around the multiprocess creation so that workers don't spawn indefinitely: