s3committer | Hadoop output committers for S3

by rdblue Java Version: Current License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | s3committer Summary

s3committer is a Java library typically used in Big Data, Docker, Spark, Amazon S3, Hadoop applications. s3committer has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. However s3committer has 4 bugs. You can download it from GitHub.

Hadoop output committers for S3

Support

Quality

Security

License

Reuse

Support

s3committer has a low active ecosystem.

It has 106 star(s) with 43 fork(s). There are 8 watchers for this library.

It had no major release in the last 6 months.

There are 9 open issues and 3 have been closed. On average issues are closed in 202 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of s3committer is current.

Quality

s3committer has 4 bugs (0 blocker, 0 critical, 1 major, 3 minor) and 76 code smells.

Security

s3committer has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

s3committer code analysis shows 0 unresolved vulnerabilities.

There are 1 security hotspots that need review.

License

s3committer is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

s3committer releases are not available. You will need to build from source code and install.

Build file is available. You can build the component from source.

Installation instructions are available. Examples and code snippets are not available.

s3committer saves you 1319 person hours of effort in developing the same functionality from scratch.

It has 2958 lines of code, 222 functions and 19 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed s3committer and discovered the below as its top functions. This is intended to give you an instant insight into s3committer implemented functionality, and help decide if they suit your requirements.

Performs the commit
Commits a pending job
Gets a list of pending upload files
Returns the final output path
Commit the task
Obtain the list of files on the local filesystem
Creates a multipart upload request
Commit a task
Attempts to delete the pending file
Generate a random temp dir
Performs commit
Waits for all futures to complete
Throw one or more exceptions
This method determines whether a file is on the local filesystem
Sets up directories

Get all kandi verified functions for this library.

s3committer Key Features

No Key Features are available at this moment for s3committer.

s3committer Examples and Code Snippets

No Code Snippets are available at this moment for s3committer.

Community Discussions

Trending Discussions on s3committer

Unable to get S3A Directory Committers to write files in Spark 3.0.0

QUESTION

Unable to get S3A Directory Committers to write files in Spark 3.0.0

Asked 2020-Jul-02 at 15:10

We are using Spark 3.0.0 and we are trying to write to S3a using the new S3A committers that Ryan Blue at Netflix wrote and were added in Spark by steveloughran.

We are using the build without Hadoop (spark-3.0.0-bin-without-hadoop) and provide our own Hadoop Jars (Hadoop 3.2.1).

The original issue I was facing was that we were getting a class not found exception for org.apache.spark.internal.io.cloud.PathOutputCommitProtocol

Full trace below:

...

ANSWER

Answered 2020-Jul-02 at 15:10

This surfaces when you have > 1 machine in the spark cluster but you aren't using a shared filesystem to propagate the data about pending commits into the final dir.

make sure that fs.s3a.committer.staging.tmp.path points to something in HDFS, not paths local to the machines

Not using HDFS? well, you'd better make sure s3guard is on (for consistent s3 listings), then I'd switch to the magic committer which is pure S3 -no need for any cluster FS. Do not attempt to use it without S3Guard unless you like invalid answers

w.r.t why no spark-hadoop-cloud artifact? didn't get built in the release. The fact it adds the entire AWS SDK to the download is probably a factor. You can build it yourself though -it is probably safer to do that than mix spark artifacts

Source https://stackoverflow.com/questions/62685633

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install s3committer

This project uses gradle. To build and test, run gradle build.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: