ffbase | Basic functionality for R package ff

by edwindj R Version: Current License: No License

X-Ray Key Features Code Snippets Community Discussions(4)Vulnerabilities Install Support

kandi X-RAY | ffbase Summary

ffbase is a R library. ffbase has no bugs, it has no vulnerabilities and it has low support. You can download it from GitHub.

R is an excellent statistical tool. However its important data objects are memory objects: all processing in R takes place in memory. ff is a R package for working with vectors that are bigger than memory, but lacks at the moment some standard statistical methods. The intention of ffbase is to provide the basic statistical functions for ff objects, so programming with ff will be easier.

Support

Quality

Security

License

Reuse

Support

ffbase has a low active ecosystem.

It has 32 star(s) with 12 fork(s). There are 10 watchers for this library.

It had no major release in the last 6 months.

There are 18 open issues and 36 have been closed. On average issues are closed in 29 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of ffbase is current.

Quality

ffbase has 0 bugs and 0 code smells.

Security

ffbase has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

ffbase code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

ffbase does not have a standard license declared.

Check the repository for any license declaration and review the terms closely.

Without a license, all rights are reserved, and you cannot use the library in your applications.

Reuse

ffbase releases are not available. You will need to build from source code and install.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of ffbase

Get all kandi verified functions for this library.

ffbase Key Features

No Key Features are available at this moment for ffbase.

ffbase Examples and Code Snippets

No Code Snippets are available at this moment for ffbase.

Community Discussions

Trending Discussions on ffbase

Basic example not working for ffwhich from the ffbase-package

biglm - Error: $ operator is invalid for atomic vectors

How to use biglm with more than 2^31 observations

Reassigning values to columns in ffdf [R]

QUESTION

Basic example not working for ffwhich from the ffbase-package

Asked 2019-Sep-19 at 02:57

I'm trying to use OHDSI:s version of the SelfControlledCaseSeries package, which utilizes the ff package to handle big data. But something is not working with the ffwhich function. Running the following example, provided in the ffwhich documentation:

...

ANSWER

Answered 2019-Sep-19 at 02:57

A similar error was reported on the package's git hub. Appears to be an issue with operating system (Windows 10?). @jwijffels provides the reason in the comments:

Haven't got windows 10 machine myself but the problem clearly comes from ff::chunk, namely from ff::chunk.ff_vector which is defined as follows

The relevant part is this: b <- BATCHBYTES%/%RECORDBYTES. This calculation apparently on your machine gives 23058430092136940 for reasons beyond my understanding (given that you report it works on Rgui but not on RStudio).

You could probably get around on this by changing option ffbatchbytes to something like this options(ffbatchbytes = 84882227) - which is the number I have on my oldskool windows 7

I was able to reproduce your error and correct it using the above suggestion:

Source https://stackoverflow.com/questions/57992267

QUESTION

biglm - Error: $ operator is invalid for atomic vectors

Asked 2019-May-09 at 10:49

I am trying to run a generalized linear model on a very large dataset (several million rows). R doesn't seem able to handle the analysis, however, as I keep getting memory allocation errors (unable to allocate vector of size...etc.).

The data fit in RAM, but seem to be too large to estimate complex models. As a solution, I'm exploring using the ff package to replace r's in-RAM storage mechanism with on-disk storage.

I have successfully (I think) off-loaded the data to my hard drive, but when I attempt to estimate the glm (via the biglm package) I get the following error:

...

ANSWER

Answered 2019-May-09 at 10:49

You are using the wrong family argument.

Source https://stackoverflow.com/questions/56049950

QUESTION

How to use biglm with more than 2^31 observations

Asked 2017-Sep-16 at 00:42

I am working with a large set of data that contains more than 2^31 observations. The actual number of observations is close to 3.5 billion observations.

I am using the R package "biglm" to run a regression with approximately 70 predictors. I read in the data one million rows at a time and update the regression results. The data have been saved in the ffdf format using the R library "ffdf" to load quickly and avoid using up all my RAM.

Here is the basic outline of the code I am using:

...

ANSWER

Answered 2017-Jul-04 at 06:28

I believe that I have found the source of the issue in the biglm code.

The number of observations (n) is stored as an integer, which has a max value of 2^31 - 1.

The numeric type is not subject to this limit, and, as far as I can tell, can be used instead of integers to store n.

Here is a commit on github showing how to fix this problem with one additional line of code that converts the integer n to a numeric. As the model is updated, the number of rows in the new batch is added to the old n, so the type of n remains numeric.

I was able to reproduce the error described in this question and verify that my fix works with this code:

(WARNING: This consumes a large amount of memory, consider doing more iterations with a smaller array if you have tight memory constraints)

Source https://stackoverflow.com/questions/44486842

QUESTION

Reassigning values to columns in ffdf [R]

Asked 2017-Apr-06 at 11:49

I am having trouble doing the following operations in a larger dataset. I wonder if there is a built in way to do it with either ff or ffdf.

Example: Modifying a character columns in an ffdf object using substr and reassign it as a different column:

...

ANSWER

Answered 2017-Apr-06 at 11:49

require(ffbase)
data(iris, package = "datasets")
x <- as.ffdf(iris)
x$spec <- with(x[c("Species")], substr(Species, 1, 4))

Source https://stackoverflow.com/questions/43219031

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install ffbase

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: