initialization-actions | Run in all nodes of your cluster before the cluster starts

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

1. For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.

2. For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

It seems that in this version the type of metadata parameter is no longer dict. From the docs:

metadata (Sequence[Tuple[str, str]]) -- Additional metadata that is provided to the method.

Try with:

the following dag is working as expected, changed:

• the cluster name (cluster_name -> cluster-name).
• path for scripts.
• Dag definition.

The answer you linked is the correct way to do it: How can I include additional jars when starting a Google DataProc cluster to use with Jupyter notebooks?

If you also post the command you tried with the escaping syntax and the resulting error message then others could more easily verify what you did wrong. It looks like you're specifying an additional spark property in addition to your list of jars spark:spark.driver.memory=3000m, and tried to just space-separate that from your jars flag, which isn't allowed.

Per the linked result, you'd need to use the newly assigned separator character to separate the second spark property:

It looks like your application is depending on the BigQuery connector, not the GCS connector (bigquery.DefaultSource).

The GCS connector should always be included in the HADOOP classpath by default, but you will have to manually add the BigQuery connector jar to your application.

Assuming this is a Spark application, you can set the Spark jar property to pull in the bigquery connector jar from GCS at runtime: spark.jars='gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar'

For more installation options, see https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/master/README.md

Dataproc installs by default pyarrow 0.15.0 while the bigquery-storage-api needs a more recent version. Manually setting pyarrow to 3.0.0 at install solved the issue. That being said, PySpark has a compability setting for Pyarrow >= 0.15.0 https://spark.apache.org/docs/3.0.0-preview/sql-pyspark-pandas-with-arrow.html#apache-arrow-in-spark I've taken a look at the release notes of dataproc and this env variable is set as default since May 2020.

This is described in the documentation gcloud dataproc workflow-templates add-job pyspark:

You can pass them via the --properties option:

--properties=[PROPERTY=VALUE,…] List of key value pairs to configure Spark. For a list of available properties, see: https://spark.apache.org/docs/latest/configuration.html#available-properties.

Example using gcloud command:

Just figured it out, the format needs to be: