spark-jobserver | REST job server for Apache Spark

by spark-jobserver Scala Version: v0.11.1 License: Non-SPDX

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | spark-jobserver Summary

spark-jobserver is a Scala library typically used in Big Data, Spark applications. spark-jobserver has no bugs, it has no vulnerabilities and it has medium support. However spark-jobserver has a Non-SPDX License. You can download it from GitHub.

spark-jobserver provides a RESTful interface for submitting and managing Apache Spark jobs, jars, and job contexts. This repo contains the complete Spark job server project, including unit tests and deploy scripts. It was originally started at Ooyala, but this is now the main development repo. Other useful links: Troubleshooting, cluster, YARN client, YARN on EMR, Mesos, JMX tips. Also see Chinese docs / 中文.

Support

Quality

Security

License

Reuse

Support

spark-jobserver has a medium active ecosystem.

It has 2820 star(s) with 1014 fork(s). There are 219 watchers for this library.

It had no major release in the last 12 months.

There are 105 open issues and 714 have been closed. On average issues are closed in 673 days. There are 3 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of spark-jobserver is v0.11.1

Quality

spark-jobserver has 0 bugs and 0 code smells.

Security

spark-jobserver has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

spark-jobserver code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

spark-jobserver has a Non-SPDX License.

Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

Reuse

spark-jobserver releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

It has 22334 lines of code, 1097 functions and 242 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of spark-jobserver

Get all kandi verified functions for this library.

spark-jobserver Key Features

No Key Features are available at this moment for spark-jobserver.

spark-jobserver Examples and Code Snippets

No Code Snippets are available at this moment for spark-jobserver.

Community Discussions

Trending Discussions on spark-jobserver

Monitoring for the spark-jobserver

QUESTION

Monitoring for the spark-jobserver

Asked 2022-Feb-22 at 23:18

I'm currently working with spark-jobserver, and when the spark-jobserver goes down my app just stop working but I don't get notified

There is a health check to spark-jobserver?

...

ANSWER

Answered 2022-Feb-22 at 23:18

If the service goes down, it also wouldn't notify. Therefore, there isn't a built-in option for this.

Ideally you'd use external monitoring like Prometheus Blackbox exporter, Nagios, or simply cron to curl/netcat your service.

Source https://stackoverflow.com/questions/71229476

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install spark-jobserver

The easiest way to get started is to try the Docker container which prepackages a Spark distribution with the job server and lets you start and deploy it. NOTE: Spark Job Server can optionally run SparkContexts in their own, forked JVM process when the config option spark.jobserver.context-per-jvm is set to true. This option does not currently work for SBT/local dev mode. See Deployment section for more info.
Build and run Job Server in local development mode within SBT. NOTE: This does NOT work for YARN, and in fact is only recommended with spark.master set to local[*]. Please deploy if you want to try with YARN or other real cluster.
Deploy job server to a cluster. There are two alternatives (see the deployment section): server_deploy.sh deploys job server to a directory on a remote host. server_package.sh deploys job server to a local directory, from which you can deploy the directory, or create a .tar.gz for Mesos or YARN deployment.
EC2 Deploy scripts - follow the instructions in EC2 to spin up a Spark cluster with job server and an example application.
EMR Deploy instruction - follow the instruction in EMR
When POSTing new binaries, the content-type header must be set to one of the types supported by the subclasses of the BinaryType trait. e.g. "application/java-archive", "application/python-egg" or "application/python-wheel". If you are using curl command, then you must pass for example "-H 'Content-Type: application/python-wheel'".

Support

Contributions via Github Pull Request are welcome. Please start by taking a look at the contribution guidelines and check the TODO for some contribution ideas. Profiling software generously provided by. YourKit supports open source projects with its full-featured Java Profiler.

Find more information at: