cloud-storage | Worker allows you to put things | Storage library

The Problem:

I am using Docker Compose to create two containers: One with a Postgres database on it and the other with Flyway on it. The goal is to use Flyway to migrate scripts to the Postgres database instance. When I run docker-compose up I get the following error:

Unable to obtain connection from database (jdbc:postgresql://db:5432/) for user 'luke_skywalker': The connection attempt failed.

My code is below and thank you for your help!

Here is my docker-compose.yml:

We are using conda to maintain a python environment and I'd like to understand why google-cloud-bigquery==1.22.0 is being installed when the latest available version is https://pypi.org/project/google-cloud-bigquery/2.16.1/ and the latest vaailable version on conda-forge (https://anaconda.org/conda-forge/google-cloud-bigquery) is 2.15.0

Here's a Dockerfile that builds our conda environment:

We've a Java 8 standalone applicat that reads from Cassandra tables, The client version we're currently using is 3.4.0. The application should also support reading from Google Cloud Storage, but once we added the GCS dependencies to the pom file we started see exceptions when reading from Cassandra. Seems like the 3.4 driver uses Guava 19, and the GCS uses Guava 30. Is it possible to make them both live together in the same Java process? Trying to exlude Guava from the cassandra-driver-core 3.4 causing the following error:

When running (new Storage()).bucket('my-bucket-name').getFiles(), I get a list of objects with this structure. All the items are set to public and I'd rather not process the objects to piece together the public urls by "hand" (https://storage.cloud.google.com/[object.metadata.bucket]/[object.metadata.name]) and I was wondering if the NodeJs client for GCP offers anything like this.

I found a similar link here except for python.

Thank you!

To speed up my cluster instantiation time, I've created a custom image with all the additional dependencies installed using miniconda3 available for dataproc image 1.5.34-debian10. (I followed the steps here: GCP Dataproc custom image Python environment to ensure I used the correct python environment).

However, when I start my cluster with --optional-components ANACONDA,JUPYTER my custom dependencies are removed and I'm left with a base installation of anaconda and jupyter. I assume the anaconda installation is overwriting my custom dependencies. Is there any way to ensure my dependencies aren't overwritten? If not, is it possible to install anaconda and jupyter as part of my custom dataproc image instead?

I've used the following command to create the custom image:

Since the beginning of this year our python dataflow jobs result in an error on worker startup:

Received an import error after upgrading to airflow2.0.2-python3.7 image. Package seems to be installed, not sure what is causing the issue and how to fix it. Tried to uninstalling and reinstalling the packages but that does not work either.

I am using the google/cloud-storage package in an API and successfully uploading pdf files to a Google Cloud Storage bucket. However, the pdf files are first saved locally before they are uploaded to the Google Cloud Storage bucket.

How can I skip saving them locally and instead upload them directly to the Google Cloud Storage bucket? I am planning to host the API on Google App Engine. This is the post for it.

This is what I am doing currently:

I am trying to run a simple spark script in a dataproc cluster, that needs to read/write to a gcs bucket using scala and the java Cloud Storage Client Libraries. The script is the following: