disk.frame | Fast Disk-Based Parallelized Data Manipulation Framework

I'm currently trying to write a function that filters some rows of a disk.frame object using regular expressions. I, unfortunately, run into some issues with the evaluation of my search string in the filter function. My idea was to pass a regular expression as a string into a function argument (e.g. storm_name) and then pass that argument into my filtering call. I used the %like% function included in {data.table} for filtering rows.

My problem is that the storm_name object gets evaluated inside the disk.frame. However, since the storm_name is only included in the function environment, but not in the disk.frame object, I get the following error:

According to the article https://diskframe.com/articles/ingesting-data.html a good use case for inmapfn as part of csv_to_disk_frame(...) is for date conversion. In my data I know the name of the date column at runtime and would like to feed in the date to a convert at read in time function. One issue I am having is that it doesn't seem any additional parameters can be passed into the inmapfn argument beyond the chunk itself. I can't use a hardcoded variable at runtime as the name of the column isn't known until runtime.

To clarify the issue is that the inmapfn seems to run in its own environment to prevent any data races/other parallelisation issues but I know the variable won't be changed so I am hoping there is someway to override this as I can make sure that this is safe.

I know the function I am calling works when called on an arbitrary dataframe.

I have provided a reproducible example below.

I'm getting this error when trying to import CSVs using this code:

some.df = csv_to_disk.frame(list.files("some/path"))

Error in split_every_nlines(name_in = normalizePath(file, mustWork = TRUE), : Expecting a single string value: [type=character; extent=3].

I got a temporary solution with a for loop that iterated through each of the files and then I rbinded all the disk frames together.

I pulled the code from the ingesting data doc

I have a disk frame that I've saved into a file. It's made up of ten chunks.

I coded every one of the columns as a character because I intend on combining these individual disk frames into one large disk frame and setting the column types at that point.

I wanted to pull the disk frame from it's file with this code

I'm looking through the docs and I don't see a function for writing to CSV.

It appears there's a function for writing the disk frame, but it's unclear what format it gets stored in

write_disk.frame

Write a data.frame/disk.frame to a disk.frame location. If df is a data.frame then using the as.disk.framefunction is recommended for most cases

Can I use fwrite or write_csv with a disk frame?

I saved a disk frame to its output directory and then restarted my R session.

I'd like to read the existing disk frame instead of recreating it elsewhere.

How might I be able to accomplish this? My folder is called outdir.df

This is how I saved the disk frame

I'd like to convert a data frame to a disk frame and then count the first column. It's not counting the number of unique values of the column when I try it. It appears to be counting the number of workers.