clustermq | R package to send function calls | GPU library

by mschubert R Version: v0.8.95 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(4)Vulnerabilities Install Support

kandi X-RAY | clustermq Summary

clustermq is a R library typically used in Hardware, GPU applications. clustermq has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH

Support

Quality

Security

License

Reuse

Support

clustermq has a low active ecosystem.

It has 134 star(s) with 23 fork(s). There are 10 watchers for this library.

It had no major release in the last 12 months.

There are 22 open issues and 221 have been closed. On average issues are closed in 334 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of clustermq is v0.8.95

Quality

clustermq has 0 bugs and 0 code smells.

Security

clustermq has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

clustermq code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

clustermq is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

clustermq releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of clustermq

Get all kandi verified functions for this library.

clustermq Key Features

No Key Features are available at this moment for clustermq.

clustermq Examples and Code Snippets

No Code Snippets are available at this moment for clustermq.

Community Discussions

Trending Discussions on clustermq

targets stalls with tar_make_clustermq()

Reverse SSH tunnel into AWS Batch array jobs

Interprocess communication on AWS Batch

Using clustermq R package as a parallel backend for foreach

QUESTION

targets stalls with tar_make_clustermq()

Asked 2022-Apr-15 at 20:04

I have a long-ish targets pipeline (takes more than an hour to execute) for which parallel execution is possible. Specifically, many, but not all, calculations can be done in parallel across 155 countries and 60 years. There are times when the country-specific calculations are aggregated from continents to the world, for example, and that sum is not amenable to parallel execution. I am running the pipeline on my local machine only (not on a cluster or networked computer).

When I run the pipeline with 5 countries and tar_make_clustermq(workers = 8) (on an 10-core machines, 14-inch Apple silicon MacBook Pro and iMac Pro), the pipeline is successful. Furthermore, I see 5 processors in use simultaneously. However, when I run the pipeline with 6 or more countries, there are several times when the pipeline stalls or seemingly switches to single-threaded execution. I have found that I need to restart the pipeline with tar_make_clustermq(workers = 8) or (worse) restart with tar_make() (single-threaded) to get it going again.

The points in the pipeline when restart is required are 100% repeatable for a specific set of countries. The points in the pipeline when a restart is required changes with the countries in the analysis.

It would be pretty difficult to develop a reprex for this behavior, because of the large files and pipelines involved. So at this time, I am requesting suggestions for next steps for debugging or changing course altogether. Here are some specific questions:

I have searched and found only this report (https://github.com/ropensci/targets/issues/182). Have I missed other reports of similar behavior?
If others found unreliable behavior from targets and clustermq on the local machine, what hints can you provide for getting around these problems?
I have considered switching from clustermq to future in targets. I'm wondering if that switch would provide improvements. I have not tried future. So if someone has experience with both, I welcome your input.

Thanks in advance for any hints!

...

ANSWER

Answered 2022-Apr-15 at 20:04

I switched to using future with much success. I'm using future::plan(future.callr::callr). My pipeline no longer hangs/stalls, as it did when using clustermq. Rather it completes without intervention, as desired. For this pipeline, at least, future is the way to go!

Source https://stackoverflow.com/questions/71886144

QUESTION

Reverse SSH tunnel into AWS Batch array jobs

Asked 2020-Dec-10 at 07:47

What exactly would it take to reverse tunnel into an AWS Batch array job from the local computer submitting the job (e.g. via AWS CLI)? Unlike the typical reverse tunneling scenario, the remote nodes here do not share a local network with the local computer. Motivation: https://github.com/mschubert/clustermq/issues/208. Related: ssh into AWS Batch jobs.

And yes, I am aware that SSH is easier in pure EC2, but Batch is preferable because of its support for arbitrary Docker images, easy job monitoring, and automatic spot pricing.

...

ANSWER

Answered 2020-Dec-10 at 07:47

Use a Unmanaged Compute Environment. Then you can ssh into your ec2 instances as you normally would, as they are under your control. A managed compute environment means that your use of ec2 is abstracted away from you, so you cannot ssh into the underlying instances. To find out what instance a job is running on, you can use the metadata endpoint.

Source https://stackoverflow.com/questions/65066533

QUESTION

Interprocess communication on AWS Batch

Asked 2020-Nov-13 at 08:57

Is it possible to use ZeroMQ for interprocess communication between a job on AWS Batch and a local parent process? If so, is there a simple example I can refer to? I am having trouble finding documentation on this.

I am asking with hopes that the clustermq R package will someday support an AWS Batch backend: https://github.com/mschubert/clustermq/issues/208. clustermq can already send R function calls to workers on traditional job schedulers like SLURM and communicate with the workers using ZeroMQ.

...

ANSWER

Answered 2020-Nov-13 at 08:57

Q : "Is it possible to use ZeroMQ for interprocess communication between a job on AWS Batch and a local parent process?"

ZeroMQ is a can-do toolbox, so the only problem will be with the Cloud-owner dictated policies (plus the grooming-factor bottlenecks, as these always has been the new kind of singularity-related issues, a SPoF if someone wants...)

Distributed-computing per-se has no other problems with ZeroMQ.

A PoC Example

We may instantiate this code on all remote-nodes and let 'em run & communicate with your localhost, sure if not blocked by Cloud-owner's set of policies and if you can let the receiving side survive the Cloud-initiated tsunami ;o)

Connectivity was set so that remote-senders code "knows" the localhost's either public IP address, or similarly motivated SSH-tunneling meta-plane, that mediates interconnections through the SSH-port-forwarding-tunnels, thus allowing all the Cloud remote-senders (if not blocked by the Cloud-owner...) actually reach to localhost private IP address, as if they were in the same private LAN-segment.

_{( a Python code provided for the ease of cross-porting )}

A localhost-receiver mock-up example :

Source https://stackoverflow.com/questions/64787462

QUESTION

Using clustermq R package as a parallel backend for foreach

Asked 2020-Jun-05 at 14:27

I've started using the clustermq package as a parallel backend for a drake pipeline, and have been very impressed over the performance improvements that I've observed. I'm interested in evaluating the use of clustermq / rzmq in settings outside of drake, but seemingly can't get the example using foreach listed in the User Guide (in the subsection titled "As parallel foreach backend") to work. What am I missing here?

In the example below on my 4-core machine, I would expect the following code to run in close to 5 seconds, yet it runs in close to 20 seconds. When I use similar code to run some heavy processing, I'm only observing one core doing significant work.

...

ANSWER

Answered 2020-Jun-01 at 19:32

The good news is,
the R-console reported the work-units got dispatched to the 4 worker jobs (ID: 6856) ...

To see how the acounting of CPU-usage works ( as in unix.time() ), test this ( the online R-code execution were not able to load either the library(foreach) nor the library(doParallel) to show the differences live, online, yet the code ought be clear on how to do that on a properly configured localhost R system :

Source https://stackoverflow.com/questions/62134030

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install clustermq

First, we need the [ZeroMQ](https://github.com/zeromq/libzmq) system library. This is probably already installed on your system. If not, your package manager will provide it:.

Support

We use Github’s [Issue Tracker](https://github.com/mschubert/clustermq/issues) to coordinate development of clustermq. Contributions are welcome and they come in many different forms, shapes, and sizes. These include, but are not limited to:.

Find more information at: