s4cmd | Super S3 command line tool

by bloomreach Python Version: Current License: Apache-2.0

X-Ray Key Features Code Snippets(2)Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | s4cmd Summary

null

Super S3 command line tool

Support

Quality

Security

License

Reuse

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of s4cmd

Get all kandi verified functions for this library.

s4cmd Key Features

No Key Features are available at this moment for s4cmd.

s4cmd Examples and Code Snippets

Instaclone,Installation

Python

Lines of Code : 1

License : Permissive (Apache-2.0)

Copy

pip install instaclone

Instaclone,Running tests

Python

Lines of Code : 1

License : Permissive (Apache-2.0)

Copy

$ TEST_BUCKET=my-s3-bucket tests/run.sh

Community Discussions

Trending Discussions on s4cmd

Multipart upload to S3 with hash verification

Does s4cmd support signature version 4 because i am unable to upload files to s3 bucket (LONDON)

QUESTION

Multipart upload to S3 with hash verification

Asked 2019-Aug-20 at 09:13

I am looking for a command line tool or a Python library which allows uploading big files to S3, with hash verification.

There is an AWS article explaining how it can be done automatically by supplying a content-md5 header.

Yet, it is not clear which command line tools do or do not do this:

rclone's documentation states that

files uploaded with multipart upload don’t have an MD5SUM.
s3cmd doesn't say anything about this, but it supports md5 for the sync feature
s4cmd has a whole paragraph in the manual, but it's still not clear if an upload is actually verified
boto3 / s3transfer's upload_file() method doesn't really say anything

Do you have information about any of these tools, or some other tool or Python library or boto3 snippet which handles big file uploads to s3 with the reliability of rsync?

...

ANSWER

Answered 2017-May-06 at 06:41

Why not use aws cli? It has support for md5 verification, even for multipart uploads. See this. Both upload-part and put-object commands of s3api support content verification using hash.

And s4cmd does verify uploads using custom content header containing md5.

As for Etag calculation mentioned in S4cmd doc, see this.

Source https://stackoverflow.com/questions/43794838

QUESTION

Does s4cmd support signature version 4 because i am unable to upload files to s3 bucket (LONDON)

Asked 2017-Jul-06 at 08:51

I am trying to upload files to s3 bucket(LONDON) i.e. eu-west-2. S4cmd is not working.

s4cmd put /home/username/Documents/file-1.json s3://[BUCKETNAME]/file-1.json

error when i run this command is : -

...

ANSWER

Answered 2017-Jul-06 at 08:51

The aws s3 cp command in the AWS Command-Line Interface (CLI) uses multi-part upload to fully utilize available bandwidth, so it should give you pretty much the best speed possible.

Source https://stackoverflow.com/questions/44883168

Community Discussions, Code Snippets contain sources that include Stack Exchange Network