partools | Tools to aid coding in the R 'parallel ' package | Machine Learning library

by matloff R Version: Current License: No License

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | partools Summary

partools is a R library typically used in Artificial Intelligence, Machine Learning, Pytorch applications. partools has no bugs, it has no vulnerabilities and it has low support. You can download it from GitHub.

Miscellaneous utilities for parallelizing large computations. Alternative to MapReduce. File splitting and distributed operations such as sort and aggregate. "Software Alchemy" method for parallelizing most statistical methods, presented in N. Matloff, Parallel Computation for Data Science, Chapman and Hall, 2015. Includes a debugging aid.

Support

Quality

Security

License

Reuse

Support

partools has a low active ecosystem.

It has 38 star(s) with 8 fork(s). There are 12 watchers for this library.

It had no major release in the last 6 months.

There are 9 open issues and 1 have been closed. There are 1 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of partools is current.

Quality

partools has no bugs reported.

Security

partools has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

partools does not have a standard license declared.

Check the repository for any license declaration and review the terms closely.

Without a license, all rights are reserved, and you cannot use the library in your applications.

Reuse

partools releases are not available. You will need to build from source code and install.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of partools

Get all kandi verified functions for this library.

partools Key Features

No Key Features are available at this moment for partools.

partools Examples and Code Snippets

No Code Snippets are available at this moment for partools.

Community Discussions

Trending Discussions on partools

R in Parallel - Partools Package - Error with `calm()` function

QUESTION

R in Parallel - Partools Package - Error with `calm()` function

Asked 2018-Aug-06 at 19:53

I am using the partools package to run linear regressions in parallel. I am doing this using the calm() function, which is a wrapper for the package's version of R's lm().

I'm using 20 cores on a 64gb node.

I receive errors when I run the calm() function, and I've isolated the problem to a single variable: agelvl. Since partools must split a dataset into chunks (the number of chunks equaling the number of cores to be used), variables, from what I can tell, are stored as either character or integer. agelvl is stored as a character due to it's named levels, so I use factor() around it in the function.

Here's the code:

...

ANSWER

Answered 2018-Aug-06 at 19:53

According to the author of partools, this could be a scaling issue -- so that, even if no levels of a categorical variable are missing in any one chunk, the error may still occur because the number of observations in a given level are both absolutely and relatively low.

Solutions

Decrease the number of chunks: assuming there is a point at which the error will disappear, you can decrease the number of chunks; however, this also means that you are decreasing the number of cores you will use which means that (a) each chunk may be so large so that you run into memory problems or (b) the parallel processes now run too slow, or (c) both.
Alter the levels/variable structure: you can leave the desired number of chunks/cores as-is, and simply alter the levels so that each level has a critical number of observations. For agelvl, you could increase the intervals (10 years, instead of 5), or, if possible, change age from a categorical variable to a continuous one. One should keep in mind that such changes could alter the explanatory power of the model or cause the model to be incorrectly specified.

Source https://stackoverflow.com/questions/51674647

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install partools

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: