npshmex | ProcessPoolExecutor that transfers numpy arrays

npshmex is a Python library. npshmex has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can install using 'pip install npshmex' or download it from GitHub, PyPI.

Npshmex provides a drop-in replacement for concurrent.futures.ProcessPoolExecutor, using shared memory provided by [SharedArray] (rather than pickle) to transfer numpy arrays between processes. The last two lines take about ~290 ms on my laptop, but ~1250 ms using concurrent.futures.ProcessPoolExecutor: more than a factor four difference. To run this example, npshmex claims only half as much memory as the standard library ProcessPoolExecutor, based on the threshold at which I get a MemoryError. For the trivial add_one task, multiprocessing overhead is dominant even when spawning a single child process (a bare add_one(big_data) takes ~55 ms). However, since part of the multiprocessing overhead is in the parent process, it will also bottleneck more complex tasks when they are scaled over enough processes.