vroom | Vehicle Routing Open-source Optimization Machine | Robotics library

describe shouldn't be inside your Car class.

As it stands, nodejs would expect some class method which would be describe(text, callback). Instead of "text" you're supplying a string.

So this is how it should look like:

You can list files including those in all subdirectories (Data in particular) using list.files(pattern = "*.csv", recursive = TRUE)

Best practices

• Have one directory of raw and only raw data (the stuff you measured)
• Have another directory of external data (e.g. reference data bases). This is something you do can remove afterwards and redownload if required.
• Have another directory for the source code
• Put only the source code directory under version control plus one other file containing check sums of the raw and external data to proof integrity
• Every other thing must be reproducible using raw data and the source code. This can be removed after the project. Maybe you want to keep small result files (e.g. tables) which take long time to reproduce.

You can ignore the RStudio warnings, they are based on a heuristic, and in this case it’s very imprecise and misleading.

However, there are some errors in your code.

Firstly, you’re only sourceing the code if the function was already defined. Surely you want to do it the other way round: source the code if the function does not yet exist. But furthermore that check is completely redundant: if you haven’t sourced the code which defines the function yet, the function won’t be defined. The existence check is unnecessary and misleading. Remove it:

As already mentioned in the comments, the solution is rather straight-forward and the only thing necessary is to include the locale() option to the vroom call. Possible options to the locale option can be found in its documentation.

It seems like the processing time for the rowwise approach explodes quadratically:

Pivoting longer makes the calculation about 300x faster. For 50k rows, the code below took 1.2 seconds, compared to 372 seconds for the rowwise approach.

You have a very old version of R. You should update to the current version if you want things to go smoothly.

If you can't do that, here's what you'll have to do:

• Install the version of Rtools suitable for R 3.5.x.
• Install the packages you want. Some of them won't work, because they will need later versions of R, but won't declare that. So when you find one that fails, try installing the previous version of that package. If that also fails, try an even earlier one. Etc. Your R version was current in 2018, so you may need to go that far back in time to find compatible packages.
• Once you finally have everything working, try to update your packages. Maybe some of them could be more recent versions, maybe not. Do them one at a time. Typically tidyverse needs about 90 of them, so this will take a while.

So I recommend that you update your R version.

This is prone to opinion and context, to be honest, but some thoughts:

1. Return the original data. This spirit is conveyed in most of the tidyverse verb-functions and many other packages (and some in base R). If you're using either the %>% or |> pipes, doing this allows processing on the data after your function, which might be very convenient.

I haven't been able to figure out vroom solution for very large more-than-RAM (gzipped) csv files. However, the following approach has worked well for me and I'd be grateful to know about approaches with better querying speed while also saving disk space.

1. Use split sub-command inxsv from https://github.com/BurntSushi/xsv to split the large csv file into comfortably-within-RAM chunks of say, 10^5, lines and save them in a folder.
2. Read all chunks using data.table::fread one-by-one (to avoid low-memory error) using a for loop and save all of them into a folder as compressed parquet files using arrow package which saves space and prepares the large table for fast querying. For even faster operations, it is advisable to re-save the parquet files partitioned by the fields by which you need to frequently filter.
3. Now you can use arrow::open_dataset and query that multi-file parquet folder using dplyr commands. It takes minimum disk space and gives the fastest results in my experience.

I use data.table::fread with explicit definition of column classes of each field for fastest and most reliable parsing of csv files. readr::read_csv has also been accurate but slower. However, auto-assignment of column classes by read_csv as well as the ways in which you can custom-define column classes by read_csv is actually the best - so less human-time but more machine-time - which means that it may be faster overall depending on scenario. Other csv parsers have thrown errors for the kind of csv files that I work with and waste time.

You may now delete the folder containing chunked csv files to save space, unless you want to experiment loop-reading them with other csv parsers.

Other previously successfully approaches: Loop read all csv chunks as mentioned above and save them into:

1. a folder using disk.frame package. Then that folder may be queried using dplyr or data.table commands explained in the documentation. It has facility to save in compressed fst files which saves space, though not as much as parquet files.
2. a table in DuckDB database which allows querying with SQL or dplyr commands. Using database-tables approach won't save you disk space. But DuckDB also allows querying partitioned/un-partitioned parquet files (which saves disk space) with SQL commands.

EDIT: - Improved Method Below

I experimented a little and found a much better way to do the above operations. Using the code below, the large (compressed) csv file will be chunked automatically within R environment (no need to use any external tool like xsv) and all chunks will be written in parquet format in a folder ready for querying.

The progress bar is shown by readr package by default. You can disable them using options.