furrr | Apply Mapping Functions in Parallel using Futures | Development Tools library

I have access to a large computing cluster with many nodes each of which has >16 cores, running Slurm 20.11.3. I want to run a job in parallel using furrr::future_pmap(). I can parallelize across multiple cores on a single node but I have not been able to figure out the correct syntax to take advantage of cores on multiple nodes. See this related question.

Here is a reproducible example where I made a function that sleeps for 5 seconds and returns the starting time, ending time, and the node name.

WHAT I WANT: I'm trying to fit a GAM model for classification using tidymodels on a given data.

SO FAR: I'm able to fit a logit model.

I had a function successfully using future multisession, but with an update of future received the below error, which I have not figured out how to solve:

Error: values() is defunct in future (>= 1.20.0). Use value() instead.

Please see below example (where I am not using values(), but still get the error):

I am building a custom recipes function and getting an error when I try to prep() the recipe. I get the following error:

I have a tibble that includes a list-column with vectors inside. I want to create a new column that accounts for the length of each vector. Since this dataset is large (3M rows), I thought to shave off some processing time using the furrr package. However, it seems that purrr is faster than furrr. How come?

To demonstrate the problem, I first simulate some data. Don't bother to understand the code in the simulation part as it's irrelevant to the question.

data simulation function

Below is my attempt at a minimal reproducible example. Briefly explained, I am using rollApply from the rowr package to calculate a function over a rolling window, and using data from two columns simultaneously. If possible, I would like to skip n steps between each time the function is calculated on a new window. I will try to make it clear what I mean in the example below.

Here is the example data:

In the R language, optimization can be achieved by using purrr::map() or furrr::future_map() functions. However, I am not sure how does optimization works for np.array() methods. Indeed, I would like to understand how does Python and R scales out to parallel processing [1, 2] in terms of complexity and performance.

Thus, the following questions arise:

How does the optimization of np.array() in Python works comparing to purrr::map() and furrr::future_map() functions in the R language?

By doing a simple tictoc test on purrr/furrr, I can observe that we have a big win from vectorization in both cases. Nonetheless, I can also notice that the results seem to show that the R language is just fundamentally faster.

I noticed that when loading furrr after raster, I am warned that values is masked:

From the data.table package website, given that:

"many common operations are internally parallelized to use multiple CPU threads"

• I would like to know if that is the case when Map() is used within a data.table?

The reason for asking is because I have noticed that comparing the same operation on a large dataset (cor.test(x, y) with x = .SD and y being a single column of the dataset), the one using Map() performs quicker than when furrr::fututre_map2() is used.