parallel-process | : running : Run multiple | BPM library

I'm new to parallel-processing and attempting to parallelize a for loop in which I create new columns in a data frame by matching a column in said data frame with two other data frames. j, the data frame I'm attempting to create columns in is 400000 x 54. a and c, the two data frames I'm matching j with are 5000 x 12 and 45000 x 8 (respectively).

Below is my initial loop prior to the attempt at parallelizing:

My question is how can I utilize multiple cores of my iMac in order to make gganimate go faster. There is another question (and more linked to below) that asks this same thing—my question is about an answer to this question: Speed Up gganimate Rendering.

In that answer, Roman and mhovd point out an example from this GitHub comment (see also this GitHub post):

Say I have a large amount of files in a list, like this

I am a beginner in parallel-processing, I want to send one value from a process belonging to communicator A to all processes in a communicator B , I tried to use the MPI_Bcast(), the process_sender should belong to Communicator B but it doesn't

I am confused why this code seems to hang and do nothing? As I try to experiment, it seems that I cannot get functions to access the queue from outside the function that added the item to the queue. Sorry, I am pretty novice. Do I need to pip install something? Update: Win10, Python 3.7.8, and this problem seems to apply to other variables besides queues.

This code works:

I am trying to scrape a bunch of urls using Selenium and BeautifulSoup. Because they are thousands and the processing that I need to do is complex and uses a lot of CPU, I need to do multiprocessing (as opposed to multithreading).

The problem right now is that I am opening and closing a Chromedriver instance once for each URL, which adds a lot of overhead and makes the process slow.

What I want to do is instead have a chromedriver instance for each subprocess, only open it once and keep it open until the subprocess finishes. However, my attempts to do it have been unsuccessful.

I tried creating the instances in the main process, dividing the set of URLS into the number of processes and sending each subprocess its subset of urls and a single driver as arguments, so that each subprocess would cycle through the urls that it got. But that didn't run at all, it did not give either results or error.

A solution similar to this with multiprocessing instead of threading got me a recursion-limit error (changing the recursion limit using sys would not help at all).

What else could I do to make this faster?

Below are the relevant parts of the code that actually works.

I'm trying to learn Golang. I know the differences between parallelism and concurrency. I'm looking for how achieve parallelism in Go. Before looking into Go I expected goroutines to be parallel, but old documentation appears to say otherwise. The setting GOMAXPROCS allows us to configure the number of threads that the application can use to run parallely. Since 1.5 GOMAXPROCS is set to the number of cores. Does that mean that since version 1.5 goroutines are inherently parallel!?

Every question I find on sites like StackOverflow look mostly outdated to me, and don't take into account this change in version 1.5. See: Parallel processing in golang

I’m also confused because the following code doesn't achieve parallelism in Go 1.10: https://play.golang.org/p/24XCgOf0jy5 (edit: to clarify, I’m not running the code in the golang playground)

Setting up GOMAXPROCS to 2 doesn't change the result, I get a concurrent program instead of a parallel one.

I'm running all my tests on an 8 core system.

Edit: For future reference:

I got carried away by this blog: https://www.ardanlabs.com/blog/2014/01/concurrency-goroutines-and-gomaxprocs.html where parallelism is achieved without much hassle in a small for-loop. The answer made by @peterSO is completely valid, for some reason, in Go 1.10 I couldn't replicate the results of the blog. A great comment was made that corrected my interpretation of the change done in 1.5. In 1.5 the default value of GOMAXPROCS changed, making Go parallel by default, but the setting was already avaiable, so goroutines were already inherently parallel if you configured it properly beforehand pre-1.5.

I have a python code that performs filtering on a matrix. I have created a C++ interface using pybind11 that successfully runs in serialized fashion (please see the code in below).

I am trying to make it parallel-processing to hopefully reduce the computation time compared to its serialized version. To do this, I have splitted my array of size M×N into three sub-matrices of size M×(N/3) to process them in parallel using the same interface.

I used ppl.h library to make a parallel for-loop and in each loop call the python function on a sub-matrix of size M×(N/3).

I copied an expample code from a python multiprocessing tutorial and slightly modified it: