s3-stream-upload | Manages streaming of data to AWS S3

by alexmojaki Java Version: Current License: MIT

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | s3-stream-upload Summary

s3-stream-upload is a Java library. s3-stream-upload has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub, Maven.

This library allows you to efficiently stream large amounts of data to AWS S3 in Java without having to store the whole object in memory or use files. The S3 API requires that a content length be set before starting uploading, which is a problem when you want to calculate a large amount of data on the fly. The standard Java AWS SDK will simply buffer all the data in memory so that it can calculate the length, which consumes RAM and delays the upload. You can write the data to a temporary file but disk IO is slow (if your data is already in a file, using this library is pointless). This library provides an OutputStream that packages data written to it into chunks which are sent in a multipart upload. You can also use several streams and upload the data in parallel. The entrypoint is the class StreamTransferManager. Read more in the javadoc, including a usage example. This is available from maven central.

Support

Quality

Security

License

Reuse

Support

s3-stream-upload has a low active ecosystem.

It has 190 star(s) with 57 fork(s). There are 11 watchers for this library.

It had no major release in the last 6 months.

There are 1 open issues and 26 have been closed. On average issues are closed in 49 days. There are 2 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of s3-stream-upload is current.

Quality

s3-stream-upload has 0 bugs and 0 code smells.

Security

s3-stream-upload has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

s3-stream-upload code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

s3-stream-upload is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

s3-stream-upload releases are not available. You will need to build from source code and install.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

s3-stream-upload saves you 2297 person hours of effort in developing the same functionality from scratch.

It has 5017 lines of code, 65 functions and 34 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed s3-stream-upload and discovered the below as its top functions. This is intended to give you an instant insight into s3-stream-upload implemented functionality, and help decide if they suit your requirements.

Waits for the multipart upload
Uploads a stream part
Calculates MD5 digest value
Generate MD5
Gets the output stream to write to
Submits a task
Throws an InterruptedException when interrupted
Aborts the upload
Details about this upload operation
Removes the middle of the string
Sets the number of streams to write
Ensures that the queue can be set
Puts any remaining data into the stream
Puts the current stream
Provides an iterator over the items in the queue
Aborts the task
Configures the data integrity check
Sets the number of threads to upload
Sets the queue capacity
Appends the given stream to this one
Inserts an item into the queue waiting if necessary
String representation of this OutputOutputStream
String representation of this object
Sets the size of the part

Get all kandi verified functions for this library.

s3-stream-upload Key Features

No Key Features are available at this moment for s3-stream-upload.

s3-stream-upload Examples and Code Snippets

No Code Snippets are available at this moment for s3-stream-upload.

Community Discussions

Trending Discussions on s3-stream-upload

Upload zip archive files to S3 with node

QUESTION

Upload zip archive files to S3 with node

Asked 2018-Apr-19 at 14:55

I download a ziparchive from an api, which contains gzipped files, and i need to take the gz files and save to s3. Don't want to uncompress or anything. Just move to S3.

When i open the archive, it has a folder w/ random numbers, /12345/file1.gz, and many files, /12345/file2.gz, etc.

I've tried yauzl and adm-zip, but don't understand how to take each entry in the archive and just send to s3. I have s3-stream-upload package, which i can use to send. Just can't get it right. Thanks for any help

...

ANSWER

Answered 2018-Apr-19 at 14:55

Answer was doing a straight s3 put with readStream as the body of the object...

Source https://stackoverflow.com/questions/49922937

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install s3-stream-upload

You can download it from GitHub, Maven.
You can use s3-stream-upload like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the s3-stream-upload component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: