multiprocessing_generator | Prefetch elements from a Python generator

multiprocessing_generator relies on the multiprocessing module using the 'fork' start method, because it assumes that the wrapped nested function (defined in ParallelGenerator's __init__) can be passed as the target of a multiprocessing.Process object. On a forking based platform, this is fine; the child process inherits the complete state of the parent (with a few small exceptions like threads), so it has equal access to the wrapped nested function (it inherited an exact copy after all).

Problem is, on Windows, the only available start method is 'spawn', which requires the target (and all arguments) to be pickleable (it pickles them, sends them to the child over IPC, and reconstructs them there), and nested functions are never pickleable (pickleing a function involves pickling its qualified name to be imported and used on the other side, and the qualified name of a nested function involves unimportable components, in this case, pickleing ParallelGenerator.__init__..wrapped fails because is clearly not an importable name).

Basically, multiprocessing_generator only works on UNIX-like systems, and only if you're using the default start method ('fork'); if you called set_start_method with some other value ('forkserver' or 'spawn'), multiprocessing_generator can't be made to work.

While this is a bug, it's not a particularly critical bug in most cases; there is very little benefit to the module if the generator must have its values pickled, because most such generators either aren't pickleable themselves (e.g. most file-like objects) or pickleing them involves running them to completion (in which case you've lost all your parallelism).

Sorry, but the simple answer here is: Don't use multiprocessing_generator on Windows.

That said, if your generator is I/O-bound, you might be able to benefit from the module by importing it, then immediately monkey-patching it to replace all multiprocessing components it relies on with their equivalent multiprocessing.dummy names (which are backed by threads, and don't rely on pickling), e.g.