s3 | Advanced S3 library for Go

by eaigner Go Version: Current License: MIT

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | s3 Summary

s3 is a Go library. s3 has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Simple S3 API for Go. Docs here. The configuration contains you credentials and bucket info. Object(path) returns a new S3 object handle bound to the configuration it was created from. Writing to the WriteAbortCloser returned by Writer() allows you to upload objects. Reading from the ReadCloser returned by Reader() allows you to download objects. Check if an object exists.

Support

Quality

Security

License

Reuse

Support

s3 has a low active ecosystem.

It has 20 star(s) with 4 fork(s). There are 2 watchers for this library.

It had no major release in the last 6 months.

s3 has no issues reported. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of s3 is current.

Quality

s3 has 0 bugs and 0 code smells.

Security

s3 has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

s3 code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

s3 is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

s3 releases are not available. You will need to build from source code and install.

Installation instructions are not available. Examples and code snippets are available.

It has 1628 lines of code, 52 functions and 8 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed s3 and discovered the below as its top functions. This is intended to give you an instant insight into s3 implemented functionality, and help decide if they suit your requirements.

authString returns the request s authorization string
canonicalResource returns the canonical URL and raw query string .
newS3Error returns a new s3 error .
newWriter creates a new writer .
escape string
trim returns a trimmed string

Get all kandi verified functions for this library.

s3 Key Features

No Key Features are available at this moment for s3.

s3 Examples and Code Snippets

No Code Snippets are available at this moment for s3.

Community Discussions

Trending Discussions on s3

Terraform AWS Provider Error: Value for unconfigurable attribute. Can't configure a value for "acl": its value will be decided automatically

Send argument to yml anchor for a step in bitbucket-pipelines.yml

@aws-sdk/lib-storage to Stream JSON from MongoDB to S3 with JSONStream.stringify()

Trust issue while sending a post to my API since DST Root CA X3 Expiration

kubernetes config map data value externalisation

s3fs suddenly stopped working in Google Colab with error "AttributeError: module 'aiobotocore' has no attribute 'AioSession'"

S3 Lambda Trigger not triggering for EVERY file upload

“500 Internal Server Error” with job artifacts on minio

Error installing provider "aws": openpgp: signature made by unknown entity

Lambda with SQSEvent & large batch size invokes multiple instances each handling few items

QUESTION

Terraform AWS Provider Error: Value for unconfigurable attribute. Can't configure a value for "acl": its value will be decided automatically

Asked 2022-Feb-15 at 13:50

Just today, whenever I run terraform apply, I see an error something like this: Can't configure a value for "lifecycle_rule": its value will be decided automatically based on the result of applying this configuration.

It was working yesterday.

Following is the command I run: terraform init && terraform apply

Following is the list of initialized provider plugins:

...

ANSWER

Answered 2022-Feb-15 at 13:49

Terraform AWS Provider is upgraded to version 4.0.0 which is published on 10 February 2022.

Major changes in the release include:

Version 4.0.0 of the AWS Provider introduces significant changes to the aws_s3_bucket resource.
Version 4.0.0 of the AWS Provider will be the last major version to support EC2-Classic resources as AWS plans to fully retire EC2-Classic Networking. See the AWS News Blog for additional details.
Version 4.0.0 and 4.x.x versions of the AWS Provider will be the last versions compatible with Terraform 0.12-0.15.

The reason for this change by Terraform is as follows: To help distribute the management of S3 bucket settings via independent resources, various arguments and attributes in the aws_s3_bucket resource have become read-only. Configurations dependent on these arguments should be updated to use the corresponding aws_s3_bucket_* resource. Once updated, new aws_s3_bucket_* resources should be imported into Terraform state.

So, I updated my code accordingly by following the guide here: Terraform AWS Provider Version 4 Upgrade Guide | S3 Bucket Refactor

The new working code looks like this:

Source https://stackoverflow.com/questions/71078462

QUESTION

Send argument to yml anchor for a step in bitbucket-pipelines.yml

Asked 2022-Jan-21 at 19:45

I would like to send arguments when I call an anchor with bitbucket pipelines

Here is the file I am using, I have to call after-script because I need to push to a certain S3 bucket

...

ANSWER

Answered 2022-Jan-21 at 19:45

To the best of my knowledge, you can only override particular values of YAML anchors. Attempts to 'pass arguments' won't work.

Instead, Bitbucket Pipelines provide Deployments - an ad-hoc way to assign different values to your variables depending on the environment. You'll need to create two deployments (say, dev and uat), and use them when referring to a step:

Source https://stackoverflow.com/questions/68976555

QUESTION

@aws-sdk/lib-storage to Stream JSON from MongoDB to S3 with JSONStream.stringify()

Asked 2021-Oct-07 at 18:01

I'm trying to Stream JSON from MongoDB to S3 with the new version of @aws-sdk/lib-storage:

...

ANSWER

Answered 2021-Oct-07 at 15:58

After reviewing your error stack traces, probably the problem has to do with the fact that the MongoDB driver provides a cursor in object mode whereas the Body parameter of Upload requires a traditional stream, suitable for be processed by Buffer in this case.

Taking your original code as reference, you can try providing a Transform stream for dealing with both requirements.

Please, consider for instance the following code:

Source https://stackoverflow.com/questions/69424322

QUESTION

Trust issue while sending a post to my API since DST Root CA X3 Expiration

Asked 2021-Oct-01 at 15:05

I have a C# api running on a aws S3 with ubuntu. This API is use by a website, a windows application and a xamarin app deployed on Samsung android devices.

Since today 16:00 (paris time), the android part is not working anymore, I have a "trust issue". Clearly it seems to be related with DST Root CA X3 Expiration (No release on my side and the timing is perfect).

But I don't understand why...

SSL certificate

I checked my SSL certificate and regarding let'sencrypt forums, I have one of the path base on "ISRG Root X1". The second one is base on "DST Root CA X3" (expired). I renew them anyway to be sure, but still the same certificate path. (and no problem for chrome to contact them).

Internet with https is working

I can reach internet with a webview inside the app (to my website in https)

Can't connect using restsharp

When I use RestSharp to contact my server, I have the trust issue.

My android devices are all the same: Samsung A7 tab, half up to date, the other half was update in august, all of them with Android 11. So theorically they are "not concerned" with this certificate expiration.

Can the problem come from Xamarin or RestSharp ? Maybe my server certificate ?

EDIT Ok half resolved.... If I go to the "Trusted Root Certificates folder" in my android device (don't know the exact name), If I disable the "Digital Signature Trust Co. - DST Root CA X3", it's working again !

Not a "real solution" since I need to update something like 150 devices... 2 options in my mind

Can I force RestSharp to use a certificate more than another ?
Is it just because Android know the expiration date is 30/09 and still use it because we are still the 30/09 and everythin will work Tomorow ?

EDIT 2 resolved.

Thx to all of you, sorry I should have been able to validate this answer before some post, but stackoverflow was on readonly mode this night and I fall asleep after that.

What I did (not sure if all step are mandatory).

1/ I updated the certbot since mine was < 1 (check with certbot --version)

...

ANSWER

Answered 2021-Sep-30 at 21:09

We’ve had similar issues today, unfortunately we were using older Amazon Linux on elasticbeanstalks. Upgrading to the latest Ubuntu build in your case should fix your issues.

The issue we had was the Amazon Linux version trusted certificate service was always adding the expired root certificate.

The reason restsharp is having problems is probably because it’s trying to do something like a curl request behind the scenes and is doing a handshake to verify the validity of the ssl cert when sending a request. The way it does this is checks it against certs that are trusted on the server, which includes the expired certificate.

See here for Ubuntu builds that have the latest certs upgrade https://ubuntu.com/security/notices/USN-5089-1

Source https://stackoverflow.com/questions/69397845

QUESTION

kubernetes config map data value externalisation

Asked 2021-Aug-24 at 10:11

I'm installing fluent-bit in our k8s cluster. I have the helm chart for it on our repo, and argo is doing the deployment.

Among the resources in the helm chart is a config-map with data value as below:

...

ANSWER

Answered 2021-Aug-19 at 10:05

Instead of using helm install you can use helm template ... --set ... > out.yaml to locally render your chart in a yaml file. This file can then be processed by Argo.

Docs

Source https://stackoverflow.com/questions/68729966

QUESTION

s3fs suddenly stopped working in Google Colab with error "AttributeError: module 'aiobotocore' has no attribute 'AioSession'"

Asked 2021-Aug-21 at 19:38

Yesterday the following cell sequence in Google Colab would work.

(I am using colab-env to import environment variables from Google Drive.)

This morning, when I run the same code, I get the following error.

It appears to be a new issue with s3fs and aiobotocore. I have some experience with Google Colab and library version dependency issues that I have previously solved by upgrading libraries in a particular order:

...

ANSWER

Answered 2021-Aug-20 at 17:09

Indeed, the breakage was with the release of aiobotocore 1.4.0 (today, 20 Aug 2021), which is fixed in release 2021.08.0 of s3fs, also today.

Source https://stackoverflow.com/questions/68864939

QUESTION

S3 Lambda Trigger not triggering for EVERY file upload

Asked 2021-Aug-05 at 22:24

In Python Django, I save multiple video files.

Save 1:

Long Video
Short Video

Save 2:

Long Video
Short Video

Save 3:

Long Video
Short Video

I have a lambda trigger that uses media converter to add HLS formats to these videos as well as generate thumbnails. These 3 saves are done in very short time periods between each other since they are assets to a Social Media Post object.

For some reason the S3 triggers for only some of the files.

Save 1 triggers S3 Lambda but not Save 2. Save 3 also triggers S3 Lambda.

My assumption is that the S3 trigger has some sort of downtime in between identifying new file uploads (In which case, I think the period in between these file uploads are near instant).

Is this assumption correct and how can I circumvent it?

...

ANSWER

Answered 2021-Aug-05 at 22:24

It should fire for all objects.

When Amazon S3 triggers an AWS Lambda function, information about the object that caused the trigger is passed in the events field:

Source https://stackoverflow.com/questions/68673809

QUESTION

“500 Internal Server Error” with job artifacts on minio

Asked 2021-Jun-14 at 18:30

I'm running gitlab-ce on-prem with min.io as a local S3 service. CI/CD caching is working, and basic connectivity with the S3-compatible minio is good. (Versions: gitlab-ce:13.9.2-ce.0, gitlab-runner:v13.9.0, and minio/minio:latest currently c253244b6fb0.)

Is there additional configuration to differentiate between job-artifacts and pipeline-artifacts and storing them in on-prem S3-compatible object storage?

In my test repo, the "build" stage builds a sparse R package. When I was using local in-gitlab job artifacts, it succeeds and moves on to the "test" and "deploy" stages, no problems. (And that works with S3-stored cache, though that configuration is solely within gitlab-runner.) Now that I've configured minio as a local S3-compatible object storage for artifacts, though, it fails.

...

ANSWER

Answered 2021-Jun-14 at 18:30

The answer is to bypass the empty-string test; the underlying protocol does not support region-less configuration, nor is there a configuration option to support it.

The trick is able to work because the use of 'endpoint' causes the 'region' to be ignored. With that, setting the region to something and forcing the endpoint allows it to work:

Source https://stackoverflow.com/questions/67005428

QUESTION

Error installing provider "aws": openpgp: signature made by unknown entity

Asked 2021-May-13 at 12:24

I am using terraform version 0.11.13, and this afternoon I am getting the following error in terraform init step Does it mean I've to upgrade the terraform version, is there a deprecation for this version for aws provider?

Full logs:

...

ANSWER

Answered 2021-May-03 at 13:50

The GPG key used for release signing and verification has been rotated. New releases of Terraform use this updated key for verifying official providers, and official provider releases will be signed with this key going forwards.

More about

Source https://stackoverflow.com/questions/67368339

QUESTION

Lambda with SQSEvent & large batch size invokes multiple instances each handling few items

Asked 2021-Apr-16 at 09:17

Bit of background, I'm using Serverless and .Net to create a lambda with a SQS trigger. The event trigger is set with batch size of 10k and wait time (Batch Window ie MaximumBatchingWindowInSeconds) of 30 seconds. Queue's visibility timeout is set to almost 16 minutes.

Now that I've set the lambda to reserved concurrency of only 1 and ran a test where I send 100 items to the queue & was hoping to see only one lambda invocation with exactly those 100 items.

Problem was that it separated the items in the queue and invoked the lambda five times instead, causing five packages to be created as part of the lambda's functionality instead of the one package I wanted. (FYI the lambda's output creates packages in s3 of the messages. I want to have fewer packages that are large.)

Now the question: Is this the expected behavior? and if so why is it so when I've set the queue to accumulate up to 10k items and instead it settled for 15.

According to the aws docs the lambda can grab fewer messages than the batchSize if the payload is larger than 256kb but my messages are very small and 100 messages are no where near 256kb. So that can't be the cause.

Suggestions for alternatives to dealing with this issue are also welcome, right now I'm thinking of running an event bridge scheduler that calls lambda with SQS ReceiveMessage api and creates a single package but then I also have to make sure to properly delete the queue afterwards.

I'm a bit clueless here, I'd appreciate any ideas you guys have. Thanks.

...

ANSWER

Answered 2021-Apr-13 at 14:32

I think a FIFO queue could be your solution:

"FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated."

[https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html][1]

Source https://stackoverflow.com/questions/67076207

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install s3

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: