intensifier | For making intense Slack emojis | Icon library

You could transform each list into a dictionary:

EDIT: new version after you updated the problem.

Sample data is now:

Assuming you want sentalize() to process individual elements of tLOTR, your problem is the loop:

I don't know what you want for the keys in your resulting dictionary, but assuming you want line numbers, I think this is what you might be thinking:

The multiprocessing library lets you make use of parallelism in concurrent Python code. Without multiprocessing the Python GIL tends to get in the way of true parallel execution, but you should see multiprocessing code as no different from other concurrency techniques. Basically, the biggest difference between multiprocessing and threads, is that state is shared via slow-ish IPC calls.

This means you need to carefully handle shared resources. Your current implementation doesn't do a great job of this; you have multiple concurrent tasks access shared resources without regard for what others may be doing. There are multiple opportunities for race conditions in your code, where multiple tasks can write to the same file, or where a nested data structure is updated without regard for other updates.

When you have to update shared data structures or files, you usually can pick between two options:

• Use synchronisation; anyone that needs to mutate a resource needs to obtain a shared lock first, or use some other form of synchronisation primitive to coordinate access.
• Make a single task responsible for mutating the resource. This usually involves one or more queues.

Note that you'll have to pass either of these objects (synchronisation primitives or queues) to child processes explicitly, see the programming guidelines; don't use references on an instance to share state.

For your case, I'd go with queues and dedicated tasks; your bottleneck is the data processing, writing data to disk and updating a few data structures with the results of the analysis tasks is relatively fast in comparison.

So use a single task to write to files; just put the serialised XML string together with the cat value into a dedicated queue, and have a separate task that pulls these from the queue and writes them to files. This separate task is then responsible for all file access, including opening and closing. This serialises file access and removes the possibility for race conditions and clobbered writes. If the data for these files comes in so thick and fast as to make this task a bottleneck, create tasks per target file.

Do the same for the shared data structures; send mutations to a queue, leave it to a dedicated task to merge the data. Updating proxy objects is not really suitable because their changes propagate to other processes via RPC calls, increasing the chances for race conditions, and locking won't guarantee that the data is consistent across all task processes!

For your simple example, updates to the Counter() object are not actually shared; each child process inherits a copy when it forks and updates that local copy, and the parent process will never see the changes made. So you'd use a local, new Counter() instance, and push that into a queue. A dedicated task can then receive these from the queue and update a local Counter() instance with the values by using total_counter.update(queued_counter), again ensuring that updates are serialised.

To illustrate, here is a contrived example counting Lorem Ipsum data; a series of count_words tasks do the counting, but pass the Counter() object they produce to a queue for a separate collating task to combine into a final word count. A separate logging task writes data from a logging queue to disk: