datafusion | DataFusion has now been donated to the Apache Arrow project

As a recommendation while developing operators on airflow, we should check the classes that are implementing the operators as documentation may lack some information due to versioning.

As commented, if you check CloudDataFusionStartPipelineOperator you will find that it makes use of a hook that gets the instance base on a project-id. This project-id its optional, so you can add your own project-id.

The failure_statuses parameter for the CloudDataFusionPipelineStateSensor wasn't introduced until v6.0.0 of the Google provider in Airflow. The example DAG reflects the provider with this version. Try upgrading to the latest Google provider and the example should work.

Be aware that there were some breaking changes between v5.1.0 to v6.0.0 of the provider.

Side note on looking at source code in Airflow. As of Airflow 2, releases of core Airflow and functionality related to service providers (e.g. hooks, operators, sensors for Google, Databricks, etc.) have been decoupled. This means that provider functionality can be released independent from core Airflow. Providers are released typically monthly. The main branch in Airflow reflects the latest code base but that does not mean it reflects the latest available code. To make sure you are looking at the correct code for the provider version you have installed, use tags when searching through the source code:

There is no need to install apache-airflow-backport-providers-google in Airflow 2.0+. This package actually backports Airflow 2 operators into Airflow 1.10.*. In addition, in Composer version composer-1.17.0-airflow-2.1.2 the apache-airflow-providers-google==5.0.0 package is already installed according to the documentation. You should be able to import the Data Fusion operators with the code snippet you posted as is.

However, if this is not the case, you should probably handle the conflict shown in the logs when trying to reinstall apache-airflow-providers-google==5.0.0:

This behavior is currently not supported by the Wrangler plugin. As you are already aware, Wrangler will only take a look at the first column to decode headers.

In this case, pre-processing the file to remove the first row is the easiest solution.

Which version of Datafusion are you running? Also what is the amount of memory for the Dataproc cluster? Sometimes we observe this issue when the Dataproc cluster ran out of memory. I would suggest increasing the amount of memory.

When you want Project Y (where your data fusion is in) to use resources from Project X (where the DLP is in) is to add the data fusion and compute engine service accounts of Project Y to Project X.

Notes:

• Data Fusion service account: service-xxxxxxx@gcp-sa-datafusion.iam.gserviceaccount.com
• Default compute engine service account: xxxxx-compute@developer.gserviceaccount.com

Project Y:

1. Go to IAM & Admin -> IAM
2. Click View by: "Members"
3. Tick checkbox "Include Google-provided role grants"
4. Look for service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com and (project number of Project Y)-compute@developer.gserviceaccount.com
5. Add role "DLP Administrator" for service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com

Project X:

1. Go to IAM & Admin -> IAM
2. Click Add
3. Under New Members, put service-(project number of Project Y)@gcp-sa-datafusion.iam.gserviceaccount.com
4. Grant role of "DLP Admininistrator"
5. Repeat step 2 to step 4 but this time put in (project number of Project Y)-compute@developer.gserviceaccount.com

Now that you are able to set the permissions, Go back to Project Y and update your Redact to point to Project X.

1. Go to Data Fusion -> Studio
2. Click Redact-> Properties
3. Put the template ID you created in Project X, in my sample it is "test_template"
4. Under Project ID, put the Project ID of Project X
5. Run your Data Fusion pipeline

I'm within an async context, and it feels like the compiler should know that and allow me to call .await from within the method

It is fundamentally impossible to await inside of a synchronous function whether or not you are within the context of a runtime. await's are transformed into yield points, and async functions are transformed into state machine's that make use of these yield points to perform asynchronous computations. Without marking your function as async, this transformation is impossible.

If I understand your question correctly, you have the following code:

The find and replace function of the wrangler is similar with "sed" wherein it applies regular expressions.

Period (.) matches any character except a newline character.

Here is the original data:

I tried this on my own project and here is the result when using the un-escaped period:

You need to escape the period symbol (.) so it will treat it as a regular period. Here is the result when escaping period:

As you can see, the period(.) was removed before "jpg".

If using directly the raw path (gs://test_buk/dir/), you might be getting an error when escaping special characters in the regex. That might be the reason for which you do not get any input file into the pipeline that matches your filter.

I suggest instead that you use ".*" to math the initial part (given that you are also specifying the path, no additional files in other folders will match the filter).

Therefore, I would use the following expressions depending on the group of files you want to use (feel free to change the extension of the files):

path = gs://test_buk/dir/

regex path filter = .*201909.*\.csv or .*201910.*\.csv

If you would like to know more about the regex used, you can take a look at (1)