gsutil | A command line tool for interacting with cloud storage services | Cloud Storage library

by GoogleCloudPlatform Python Version: v5.24 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | gsutil Summary

gsutil is a Python library typically used in Storage, Cloud Storage, Amazon S3 applications. gsutil has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub.

gsutil is a Python application that lets you access Google Cloud Storage from the command line. You can use gsutil to do a wide range of bucket and object management tasks, including:.

Support

Quality

Security

License

Reuse

Support

gsutil has a highly active ecosystem.

It has 820 star(s) with 337 fork(s). There are 76 watchers for this library.

It had no major release in the last 6 months.

There are 290 open issues and 476 have been closed. On average issues are closed in 194 days. There are 7 open pull requests and 0 closed requests.

It has a negative sentiment in the developer community.

The latest version of gsutil is v5.24

Quality

gsutil has 0 bugs and 0 code smells.

Security

gsutil has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

gsutil code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

gsutil is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

gsutil releases are not available. You will need to build from source code and install.

Build file is available. You can build the component from source.

Installation instructions are available. Examples and code snippets are not available.

It has 66345 lines of code, 3449 functions and 248 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed gsutil and discovered the below as its top functions. This is intended to give you an instant insight into gsutil implemented functionality, and help decide if they suit your requirements.

Performs a copy operation
Finds the matching key in boto3
Returns the decryption key based on src_obj
Sets the value for a given key
Run the command
Prints information about a bucket
Expand a URL
Creates a wildcard iterator
Runs the update command
Run a named command and handle exceptions
Constructs a destination URL
Run command
Run the rm command
Initializes multiprocessing variables
Retrieves the object media
Runs signurl command
Run the task
Runs the command
Creates the topic
Prints the progress
Patch a bucket
Lists objects in a bucket
Updates a bucket
Initialize worker threads
Collects metrics
Copies an object

Get all kandi verified functions for this library.

gsutil Key Features

No Key Features are available at this moment for gsutil.

gsutil Examples and Code Snippets

No Code Snippets are available at this moment for gsutil.

Community Discussions

Trending Discussions on gsutil

Trying to access GCS bucket but getting a `403 - Forbidden` error message

Creating signed url by impersonating service account from google function

cloud_sql_proxy gives "error reading config"... how do I fix this?

gcloud Firestore import: PERMISSION_DENIED

Google Artifact Registry: what's the alternative to artifacts.project-name.appspot.com?

Airflow scheduler does not start after Google Composer upgrade

"Unable to read file" when running any gsutil command

App Engine Python 2.7 - ImportError: cannot import name apiproxy

Kubernetespodoperator how to use cmds or cmds and arguments to run multiple commands

GCP Dataproc - cluster creation failing when using connectors.sh in initialization-actions

QUESTION

Trying to access GCS bucket but getting a `403 - Forbidden` error message

Asked 2022-Mar-15 at 06:45

I'm hoping to get help with the right permission settings for accessing my files from a Colab app.

Goal

I'd like to be able to access personal images in a CGS bucket from a Colab python notebook running the "Style Transfer for Arbitrary Styles" demo of Tensorflow.

Situation

I setup a GCS bucket, made it public, and was able to retrieve files and use them in the demo.

To avoid having the GCS bucket publicly accessible, I removed allUsers and changed to my account/email that's tied to both Colab and GCS.

That caused the following error message:

Error Messages

Exception: URL fetch failure on https://storage.googleapis.com/01_bucket-02/Portrait-Ali-02-PXL_20220105_233524809.jpg: 403 -- Forbidden

Other Approaches

I'm trying to understand how I should approach this.

Is it a URL problem?

The 'Authenticated URL' caused the above 403 error.

https://storage.cloud.google.com/01_bucket-02/Portrait_82A6118_r01.png

And the gsutil link:

gs://01_bucket-02/Portrait_82A6118_r01.png

Returned this error message:

Exception: URL fetch failure on gs://01_bucket-02/Portrait_82A6118_r01.png: None -- unknown url type: gs

Authentication setup

For IAM I have a service account in the project, as well as my user account (email: d@arrovox.com) that's tied to both the Colab and GCP accounts.

The Service Account role is Storage Admin. The Service Account has an inheritance from the Project. My user account, my email, is Storage Object Viewer

Assessment

Seems like the Authenticated URL is the right one, and it's a permissions issue.

Is this just about having the right permissions set in GCS, or do I need to call anything in the code before trying to return the image at the GCS URL?

I'd greatly appreciate any help or suggestions in how to troubleshoot this.

Thanks

doug

...

ANSWER

Answered 2022-Mar-15 at 06:45

storage.objects.get is the demand for viewing files from GCS, but it looks like your user account or email already has the right permission.

How should I know my account has the right permission?

I think there's a simple solution to figure it out.

copy your Authenticated URL
Paste on any website and search.

If your current account doesn't have the right permission, that will return @Gmail-account does not have storage.objects.get access to the Google Cloud Storage object.

Or you can visit permission of bucket details to check are your email and service over there and have the right role.

Source https://stackoverflow.com/questions/71477391

QUESTION

Creating signed url by impersonating service account from google function

Asked 2022-Mar-08 at 12:00

I was successfully able to create signed urls from my gcloud instance by running these commands :

...

ANSWER

Answered 2022-Mar-08 at 12:00

Posting John Hanley's comment and Tony Stark's comment as community wiki for visibility.

The error occurred because the --impersonate-service-account which OP used is only having the scope devstorage.read which is not enough to sign data.

The following article from John Hanley helped in troubleshooting and resolving the issue.

Source https://stackoverflow.com/questions/71380482

QUESTION

cloud_sql_proxy gives "error reading config"... how do I fix this?

Asked 2022-Mar-07 at 17:24

I have gcloud working in power shell:

...

ANSWER

Answered 2022-Mar-07 at 17:24

Looks like you don't have a local set of credentials.

You can either run gcloud auth login and try again. Or you can create a service account key and pass it to the proxy with the -credentials_file flag. See the Proxy docs for details.

Source https://stackoverflow.com/questions/71368361

QUESTION

gcloud Firestore import: PERMISSION_DENIED

Asked 2022-Mar-03 at 17:19

I am trying to import a bucket containing a Firestore database export into antoher Firebase project. I have been following this guide on how to do this.

When running the gcloud firestore import, I run into the following issue:

ERROR: (gcloud.firestore.import) PERMISSION_DENIED: Service account does not have access to Google Cloud Storage file: /bucket/EXPORT_PREFIX.overall_export_metadata. See https://cloud.google.com/datastore/docs/export-import-entities#permissions for a list of permissions needed. Error details: service-XXX@gcp-sa-firestore.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket.

I did however use the following command to grant access to the bucket:

...

ANSWER

Answered 2022-Mar-03 at 17:19

The error message appears to include the Service Account in question:

Source https://stackoverflow.com/questions/71335311

QUESTION

Google Artifact Registry: what's the alternative to artifacts.project-name.appspot.com?

Asked 2022-Mar-01 at 23:17

With Google Container Registry, one could use gsutil to list Docker layers stored within the registry. For example, this command would calculate total size (I'm assuming that's the billable size) of stored layers:

...

ANSWER

Answered 2022-Mar-01 at 23:17

For AR, you can use ListFiles to get all files in a repository. The repository size is just the sum of these file sizes and this will work for all repository types. (for docker, "Files" includes both manifests and layers)

Api docs for ListFiles are here: https://cloud.google.com/artifact-registry/docs/reference/rpc/google.devtools.artifactregistry.v1beta2#listfilesrequest ("Parent" here is the repository. I will look into making this more clear in the docs)

Source https://stackoverflow.com/questions/71311617

QUESTION

Airflow scheduler does not start after Google Composer upgrade

Asked 2022-Feb-28 at 16:12

Good morning,

After upgrade the Google Composer to the version 1.18 and Apache Airflow to the version 1.10.15 (using the auto upgrade from the composer) the scheduler does not seem to be able to start.

Airflow message: "The scheduler does not appear to be running. Last heartbeat was received 1 day ago.The DAGs list may not update, and new tasks will not be scheduled."

After get this I tried:

Restart web server gcloud beta composer environments restart-web-server
Try to restart Airflow-Scheduler: kubectl get deployment airflow-scheduler -o yaml | kubectl replace --force -f -
I looked the info of the pod: kubectl describe pod airflow-scheduler

Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 23 Feb 2022 15:59:13 +0000 Finished: Wed, 23 Feb 2022 16:04:09 +0000

So I deleted the pod and wait until it run by itself: kubectl delete pod airflow-scheduler-...
EDIT 1: The logs from the pod:

Dags and plugins are not synced yet

EDIT 2: Additional logs:

Building synchronization state... Starting synchronization... Copying gs://europe-west1-********-bucket/dags/sql/... Skipping attempt to download to filename ending with slash (/home/airflow/gcs/dags/sql/). This typically happens when using gsutil to download from a subdirectory created by the Cloud Console (https://cloud.google.com/console) / [0/1 files][ 0.0 B/ 11.0 B] 0% Done InvalidUrl Error: Invalid destination path: /home/airflow/gcs/dags/sql/

But it continues restarting alone and sometimes appears the CrashLoopBackOff so indicates that a container is repeatedly crashing after restarting

Not sure what could I do more :/.

Thanks for the help :)

...

ANSWER

Answered 2022-Feb-25 at 16:28

The problem that you are facing has to do with a problem where the resources are getting on the limits and this is not letting you start the Scheduler.

My assumptions are that this could be happening:

The limits set on the scheduler are causing the gcsfuse process to get killed, can you remove them to check if that stops the crashloop?
K8s cluster does not have enough resources for the Composer Agent to start the scheduler job, you can add resources to this.
You are getting a corrupted entry when it is starting for this. The thing that you could do with this is to restart the scheduler on your own, by using ssh to connect into the instance.

Source https://stackoverflow.com/questions/71240800

QUESTION

"Unable to read file" when running any gsutil command

Asked 2022-Feb-09 at 09:27

When I try and use any gsutil command on my google cloud instance, I get the following error:

...

ANSWER

Answered 2022-Feb-09 at 09:27

This looks like you've installed a pre-release version of the package which doesn't have the snap/google-cloud-sdk/217/platform/gsutil/VERSION file (it doesn't have the snap/google-cloud-sdk/217/platform/ folder at all) which the gsutil seems to need.
This is still a problem on the the current edge release of 370:

(gsutil) Unable to read file [/snap/google-cloud-sdk/221/platform/gsutil/VERSION]: [Errno 2] No such file or directory: '/snap/google-cloud-sdk/221/platform/gsutil/VERSION'

Removing the package with sudo snap remove google-cloud-sdk and installing the stable version with sudo snap install google-cloud-sdk --classic fixed this error for me.

Source https://stackoverflow.com/questions/70852264

QUESTION

App Engine Python 2.7 - ImportError: cannot import name apiproxy

Asked 2022-Feb-08 at 08:52

With the upgrade to Google Cloud SDK 360.0.0-0 i started seeing the following error when running the dev_appserver.py command for my Python 2.7 App Engine project.

...

ANSWER

Answered 2022-Feb-08 at 08:52

EDIT

This issue seems to have been resolved with Google Cloud SDK version 371

On my debian based system i fixed it by downgrading the app-engine-python component to the previous version

Source https://stackoverflow.com/questions/69465376

QUESTION

Kubernetespodoperator how to use cmds or cmds and arguments to run multiple commands

Asked 2022-Feb-07 at 12:15

I'm using GCP composer to run an algorithm and at the end of the stream I want to run a task that will perform several operations copying and deleting files and folders from a volume to a bucket I'm trying to perform these copying and deleting operations via a kubernetespodoperator. I'm having hardship finding the right way to run several commands using "cmds" I also tried using "cmds" with "arguments". Here is my KubernetesPodOperator and the cmds and arguments combinations I tried:

...

ANSWER

Answered 2022-Jan-28 at 14:04

For your first command you need to make sure that inside your docker you are able to reach the working directory that will allows you to find file /data/splitter-output\*.csv

["gsutil", "cp", "/data/splitter-output*.csv", "gs://my_bucket/data"]

You can test your commands on your docker image by using docker RUN so you can verify if you are providing correctly the commands.

On your second statement if you are making reference to a path inside your docker image again use run to test it. If you are referring to google storage you have to provide the full path.

["gsutil", "rm", "-r", "/input"]

Its worth to mention that ENTRYPOINT will run once container starts running as described on understand how cmd and entrypoint interact. As mention in the comment, if you look at the code cmds it replaces docker image ENTRYPOINT. It also recommends to follow the guidelines of Define a Command and Arguments for a Container

Source https://stackoverflow.com/questions/70876473

QUESTION

GCP Dataproc - cluster creation failing when using connectors.sh in initialization-actions

Asked 2022-Feb-01 at 20:01

I'm creating a Dataproc cluster, and it is timing out when i'm adding the connectors.sh in the initialization actions.

here is the command & error

...

ANSWER

Answered 2022-Feb-01 at 20:01

It seems you are using an old version of the init action script. Based on the documentation from the Dataproc GitHub repo, you can set the version of the Hadoop GCS connector without the script in the following manner:

Source https://stackoverflow.com/questions/70944833

Community Discussions, Code Snippets contain sources that include Stack Exchange Network