gcs | simple implementation of the golomb compressed sets | Compression library

It worked fine with above properties. Problem was I removed -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- from private_key earlier hence it was not working

Query result is in stderr. Try &> result.txt to redirect both stdout and stderr, or 2> result.txt to redirect stderr only.

It happened the same to me last friday. I think it has something to do with Cuda instalation in Google Colab but I don't know exactly the reason

The error says MemoryError. You must upgrade your ec2 instance to something with more memory. tensorflow is very memory hungry application.

It seems you are using an old version of the init action script. Based on the documentation from the Dataproc GitHub repo, you can set the version of the Hadoop GCS connector without the script in the following manner:

The folders don't exist on Cloud Storage, it's a blob storage, all the object are stored at the bucket level. For human readability and representation, the / are the folder separator, by convention.

Therefore, because directory doesn't exist, you can't grant any permission on it. The finer granularity is the bucket.

In your use case, you can't grant a write access at folder level, but you can create 1 bucket per user and therefore grant the impersonated service account on the bucket.

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

1. For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.

2. For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

When running jobs in cluster mode, the driver logs are in the Cloud Logging yarn-userlogs. See the doc:

By default, Dataproc runs Spark jobs in client mode, and streams the driver output for viewing as explained, below. However, if the user creates the Dataproc cluster by setting cluster properties to --properties spark:spark.submit.deployMode=cluster or submits the job in cluster mode by setting job properties to --properties spark.submit.deployMode=cluster, driver output is listed in YARN userlogs, which can be accessed in Logging.

I think the error message is clear:

Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found

You need to add the Jar file which contains the above class to SPARK_CLASSPATH

Please see Issues Google Cloud Storage connector on Spark or DataProc for complete solutions.