vroom | Elgg supercharged - A simple way to speed up elgg | Architecture library

I need to fill a lot of very large arrays by opening up thousands of csv files, extracting columns of data, and inserting them into 3D and 4D matrices. I've tried writing this in parallel, but what always happens is that my computer crashes when my memory fills up. I've looked at this question, Parallel `for` loop with an array as output, but I have not gotten those suggestions to work for me. Here's my code (generalized where needed):

The problem is that I have multiple raw data, in multiple .txt, with data like this in every file:

I'm trying to come up with a neat/fast way to read files delimited by newline (\n) characters into more than one column.

Essentially in a given input file, multiple rows in the input file should become a single row in the output, however most file reading functions sensibly interpret the newline character as signifying a new row, and so they end up as a data frame with a single column. Here's an example:

The input files look like this:

I made a function drive_car inside of the class car, but I do not know how to call it from the class it is defined in, outside of the class, while staying inside of its class.

Basically I want to be able to store a function as a field in a struct, then call that function later. Fairly simple, but I can't figure out a good way to do it.

For example:

I have many txt files that contain the same type of numerical data in columns separated by ;. But some files have column headers with spaces and some don't (created by different people). Some have extra columns which that I don't want.

e.g. one file might have a header like:

I'm trying to create a dashboard in R Shiny. As part of this dashboard, I have a very large dataset which has a column added reactively and in turn three reactive subsets of this dataset are produced by filtering on different dates.

So far, I have achieved the above via dplyr and using filter and mutate functions. However, I've noticed that it seems to be these points in the code that is slowing it down. It takes approx 10 seconds to process each table when any of the reactive variables are changed which trigger these tables to update. So I'm looking to speed this up.

I understand dplyr is much slower than data.table, but it is on the other hand, easier to understand the syntax. I'm also aware the dtplyr package exists to translate the dplyr code into data.table syntax, but I'm having trouble getting it to work.

I've been looking at the documentation and some youtube demonstrations, and as I understand it, it seems in order to use dtplyr, I need to use lazy_dt function to convert the table and then apply the standard dplyr functions to that table and then use the as.data.table() or as.tibble() command to convert it back to a data table.

However, the code isn't working, even though I've basically followed the exact same syntax as in the demonstration.

Here is some example code using the libraries I have imported in my main code, and which uses the iris dataset. It produces the following error at the print and as.data.table commands:

I am running a foreach loop where I want to read in data, modify it, and output the files in parallel. It works fine, but I always get output of a list of the files after the loop finishes. If I include rm(data) in the loop, the data is gone, but a list of empty data frames is output. Here's a generalization of my code:

I have a df where I am subtracting the differences of times of two dates using a unique ID to do so. I do get the correct output however, I have an issue with consistency in the output of the time difference. For example on row 2 a difference of 1.500 is calculated which is 90 minutes or 1 Hr and 30 minutes. On row 8 there is a difference of 20.00 calculated which is 20 minutes.

I have tried to convert this column to a standard time format but I am not successful. If I try to covert the column to minutes, times that are already in minutes are artificially higher while those in hours are correct. Any feedback anyone can provide to allow me to have a way to convert this column to a consistent format preferably minutes will be extremely helpful.