taskflow | purpose Parallel and Heterogeneous Task Programming System | Architecture library

The parallel dependency is occurring because calling the last_task() TaskFlow function and setting the task dependency to it (implicitly via the TaskFlow API) is done within the same loop which calls the other tasks. Each call of a TaskFlow function will create a new task node. If last_task was pulled outside the loops and only the necessary dependencies were set inside the loops, you would achieve the desired structure.

Let's take a simplified version of your code as an example.

Unfortunately I am not able to reproduce your issue. The similar code below parses, renders a DAG, and completes successfully on Airflow 2.0, 2,1, and 2.2:

You can just set ti in the decorator as:

1. There is virtually no difference between the two approaches - neither from logic nor performance point of view.

2. The tasks in Airflow share the data between them using XCom (https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html) effectively exchanging data via database (or other external storage). The two tasks in Airflow - does not matter if they are defined in one or many files - can be executed anyway on completely different machines (there is no task affinity in airflow - each task execution is totally separated from other tasks. So it does not matter - again - if they are in one or many Python files.

3. Performance should be similar. Maybe splitting into several files is very, very little slower but it should totally negligible and possibly even not there at all - depends on the deployment you have the way you distribute files etc. etc., but I cannot imagine this can have any observable impact.

As suggested by @Josh Fell in the comments, I had two mistakes in my DAG.

1. Wrap the data in json.dumps(data) before returning it from Get_payload.
2. Remove multiple_outputs=True from the task decorator of Get_payload.

Final code:

I tried your code and works just fine, I don't get any of the mentioned warnings. I'm running Airflow v2.1.2 using the official docker-compose setup.

I found a few issues in Airflow's repo (pr, pr) of older versions related to the messages you are recieving, but those should be solved by now. Try upgrading to latest version of Airflow, that should fix the problem.

The following is what I obtained after copy and pasting your code in my running AF:

Graph View:

Logs:

airflow dags test test_tg 2021-08-24 output:

Consider the following example, the first task will correspond to your SparkSubmitOperator task:

_get_upstream_task Takes care of getting the state of the first task from the second one, by performing a query to the metadata database:

BranchPythonOperator is expected to return task_ids

You need to change the get_tasksfunction to:

You can access the execution context with get_current_context method: