clustershell | Scalable cluster administration Python framework — Manage node sets, node groups and execute command

by cea-hpc Python Version: 1.9.2 License: No License

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | clustershell Summary

clustershell is a Python library. clustershell has no bugs, it has no vulnerabilities, it has build file available and it has low support. You can install using 'pip install clustershell' or download it from GitHub, PyPI.

ClusterShell is an event-driven open source Python library, designed to run local or distant commands in parallel on server farms or on large Linux clusters. It will take care of common issues encountered on HPC clusters, such as operating on groups of nodes, running distributed commands using optimized execution algorithms, as well as gathering results and merging identical outputs, or retrieving return codes. ClusterShell takes advantage of existing remote shell facilities already installed on your systems, like SSH. ClusterShell’s primary goal is to improve the administration of high- performance clusters by providing a lightweight but scalable Python API for developers. It also provides clush, clubak and cluset/nodeset, convenient command-line tools that allow traditional shell scripts to benefit from some of the library features.

Support

Quality

Security

License

Reuse

Support

clustershell has a low active ecosystem.

It has 365 star(s) with 81 fork(s). There are 40 watchers for this library.

It had no major release in the last 12 months.

There are 94 open issues and 338 have been closed. On average issues are closed in 545 days. There are 5 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of clustershell is 1.9.2

Quality

clustershell has 0 bugs and 0 code smells.

Security

clustershell has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

clustershell code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

clustershell does not have a standard license declared.

Check the repository for any license declaration and review the terms closely.

Without a license, all rights are reserved, and you cannot use the library in your applications.

Reuse

clustershell releases are available to install and integrate.

Deployable package is available in PyPI.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

It has 19811 lines of code, 1902 functions and 75 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed clustershell and discovered the below as its top functions. This is intended to give you an instant insight into clustershell implemented functionality, and help decide if they suit your requirements.

Run ttyloop
Run a command
Set default worker key
Bind stdin to stdin
Setup the readline parser
Return a NodeSet class
Create a NodeSet from a list
Merge the contents of other with other
Updates the contents of another node
Handle incoming messages
Set the given info key
Command line interface
Install common options
Process a received message
Build the rsh command
Set the default value of the default worker
Print a unified diff
Close the process
Build pdcp command
Main event loop
Build the command line
Start the engine
Build the ssh command
Main event loop
Parse config directory
Install display options
Raise an exception
Build the scp command
Sets the given info key

Get all kandi verified functions for this library.

clustershell Key Features

No Key Features are available at this moment for clustershell.

clustershell Examples and Code Snippets

No Code Snippets are available at this moment for clustershell.

Community Discussions

Trending Discussions on clustershell

GNU Parallel threads running function without some of its given args

QUESTION

GNU Parallel threads running function without some of its given args

Asked 2018-Oct-13 at 00:31

Running a set of GNU Parallel jobs across multiple nodes in a cluster and observing that all threads appear to be sleeping on the nodes (including the local node that called the command). Upon further inspection, found that GNU Parallel appears to be removing some args when it runs the function assigned as a job. Checking parallel --version does confirm that this is the GNU version of parallel (version 20160222).

The code in question looks like

...

ANSWER

Answered 2018-Oct-13 at 00:31

1. The sleeping processes: I answer a very similar variation of this problem in another post (see https://stackoverflow.com/a/52787273/8236733). Basically, too many threads multiplied across so many nodes, seems to be overloading the MSSQL Server endpoint DB that the bcp operation is writing into (which I think is causing some kind of timeout issue from the remote DB).

As for the missing password (-P) arg value in the ps information (shown again here)

me 135296 14.5 0.0 77596 6940 ? S 00:32 0:01

/opt/mssql-tools/bin/bcp TABLENAME in /path/to/tsv/1_16_0.tsv -D -S MyMSSQLServer -U myusername -P -d myDB -c -t \t -e /path/to/logfile

I suspect that bcp does something to hide that info from ps (so it appears just blank in the output (would have been more understandable if it appears as something like "xxxx" instead, but anyway...)). A discussion about other (conveniently for this theory, also Microsoft-made) programs doing just this kind of thing can be found here: https://unix.stackexchange.com/q/88665/260742

2. The odd arguments: In bash the \ character basically says "use the literal value next character", from another SE post (https://unix.stackexchange.com/a/146665/260742)

shall preserve the literal value of the following character, with the exception of a . ... The ... shall be removed

So even though it appears that the single, quoted variable is getting split in the given output, it really is being entered as a single argument (the \s here indicating to the function to take the literal whitespace value and not take the separated strings here as different args). Eg.

Source https://stackoverflow.com/questions/52758690

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install clustershell

You can install using 'pip install clustershell' or download it from GitHub, PyPI.
You can use clustershell like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.