multiprocess | better multiprocessing and multithreading in python | Natural Language Processing library

It happened the same to me last friday. I think it has something to do with Cuda instalation in Google Colab but I don't know exactly the reason

I had the same problem a while ago, and I recommend using Nuitka hence it supports Multiprocessing. If the problem lasts, try to use the threading library:

In short, since 3.8, CPython uses the spawn start method on MacOs. Before it used the fork method.

On UNIX platforms, the fork start method is used which means that every new multiprocessing process is an exact copy of the parent at the time of the fork.

The spawn method means that it starts a new Python interpreter for each new multiprocessing process. According to the documentation:

The child process will only inherit those resources necessary to run the process object’s run() method.

It will import your program into this new interpreter, so starting processes et cetera sould only be done from within the if __name__ == '__main__':-block!

This means you cannot count on variables from the parent process being available in the children, unless they are module level constants which would be imported.

So the change is significant.

What can be done?

If the required information could be a module-level constant, that would solve the problem in the simplest way.

If that is not possible (e.g. because the data needs to be generated at runtime) you could have the parent write the information to be shared to a file. E.g. in JSON format and before it starts other processes. Then the children could simply read this. That is probably the next simplest solution.

Using a multiprocessing.Manager would allow you to share a dict between processes. There is however a certain amount of overhead associated with this.

Or you could try calling multiprocessing.set_start_method("fork") before creating processes or pools and see if it doesn't crash in your case. That would revert to the pre-3.8 method on MacOs. But as documented in this bug, there are real problems with using the fork method on MacOs. Reading the issue indicates that fork might be OK as long as you don't use threads.

The problem with your current code is that it iterates the multiprocessed results directly, and that call will block. Fortunately there's an easy solution: use apply_async exactly as suggested in the docs. But because of how you describe the use-case here and the failure, I've adapted it somewhat. Firstly, a mock task:

Your logic starts a process wrapped within the MyClass object which itself spawns a new process via the os.system call.

When you terminate the MyClass process, you kill the parent process but you leave the 7zip process running as orphan.

Moreover, the process.terminate method sends a SIGTERM signal to the child process. The child process can intercept said signal and perform some cleanup routines before terminating. This is not ideal if you want to simulate a situation where there is no chance to clean up (a power loss). You most likely want to send a SIGKILL signal instead (on Linux).

To kill the parent and child process, you need to address the entire process group.

You need to rework your function.

Python isn’t smart enough to know which part of the code you need multiprocessed.

Most likely it’s the for loop right, you want to encrypt the files in parallel. So you can try something like this.

Define the function which needs to be run for each loop, then, create the for loop outside. Then use multiprocessing like this.

Summary: we define two sketch functions f and g from entries to sets of “sketches” such that two entries e and e′ are similar if and only if f(e) ∩ g(e′) ≠ ∅. Then we can identify merges efficiently (see the algorithm at the end).

I’m actually going to define four sketch functions, f_os, f_addr, g_os, and g_addr, from which we construct

• f(e) = {(x, y) | x ∈ f_os(e), y ∈ f_addr(e)}
• g(e) = {(x, y) | x ∈ g_os(e), y ∈ g_addr(e)}.

f_os and g_os are the simpler of the four. f_os(e) includes

• (1, e.os), if e.os is known
• (2,), if e.os is known
• (3,), if e.os is unknown.

g_os(e) includes

• (1, e.os), if e.os is known
• (2,), if e.os is unknown
• (3,).

f_addr and g_addr are more complicated because there are prioritized attributes, and they can have multiple values. Nevertheless, the same trick can be made to work. f_addr(e) includes

• (1, h) for each h in e.hostname
• (2, m) for each m in e.mac, if e.hostname is nonempty
• (3, m) for each m in e.mac, if e.hostname is empty
• (4, i) for each i in e.ip, if e.hostname and e.mac are nonempty
• (5, i) for each i in e.ip, if e.hostname is empty and e.mac is nonempty
• (6, i) for each i in e.ip, if e.hostname is nonempty and e.mac is empty
• (7, i) for each i in e.ip, if e.hostname and e.mac are empty.

g_addr(e) includes

• (1, h) for each h in e.hostname
• (2, m) for each m in e.mac, if e.hostname is empty
• (3, m) for each m in e.mac
• (4, i) for each i in e.ip, if e.hostname is empty and e.mac is empty
• (5, i) for each i in e.ip, if e.mac is empty
• (6, i) for each i in e.ip, if e.hostname is empty
• (7, i) for each i in e.ip.

The rest of the algorithm is as follows.

• Initialize a defaultdict(list) mapping a sketch to a list of entry identifiers.

• For each entry, for each of the entry’s f-sketches, add the entry’s identifier to the appropriate list in the defaultdict.

• Initialize a set of edges.

• For each entry, for each of the entry’s g-sketches, look up the g-sketch in the defaultdict and add an edge from the entry’s identifiers to each of the other identifiers in the list.

Now that we have a set of edges, we run into the problem that @btilly noted. My first instinct as a computer scientist is to find connected components, but of course, merging two entries may cause some incident edges to disappear. Instead you can use the edges as candidates for merging, and repeat until the algorithm above returns no edges.

I did some investigation, but it does not fully answer the question. I am going to post the results here in case if they help somebody else.

First, if the subprocess fails, there is no traceback. So I added the additional line to display the output of subprocesses. It should be None if no errors occur. The new code:

As already explained in this answer, id implementation is platform specific and is not a good method to guarantee unique identifiers across multiple processes.

In CPython specifically, id returns the pointer of the object within its own process address space. Most of modern OSes abstract the computer memory using a methodology known as Virtual Memory.

What you are observing are actual different objects. Nevertheless, they appear to have the same identifiers as each process allocated that object in the same offset of its own memory address space.

The reason why this does not happen in the pool is most likely due to the fact the Pool is allocating several resources in the worker process (pipes, counters, etc..) before running the task function. Hence, it randomizes the process address space utilization enough such that the object IDs appear different across their sibling processes.