CatchUp | An app for catching up on things | REST library

I believe your code has a bug in the pipeline logic.

BranchPythonOperator is expected to return the task_id to follow. In your case you have:

PostgesOperator runs SQL. Your code is querying API, generate a CSV and loading it to the DB. You can not do that with PostgesOperator.

What you can do is to replace the usage of psycopg2 with PostgresHook. The hook is a wrapper around psycopg2 that expose you functions that you can interact with. This means that, for example, you don't need to handle how to connect to Postgres on your own. Simply define the connection in Admin -> Connections and reference the connection name in the hook:

I think you are looking for is 0 0/4 * * 1-5 which will run on 4th hour from 0 through 23 on every day-of-week from Monday through Friday.

Your DAG object can be:

All right, didn't research as to why this is the case, but it seems like if I create a scripts folder inside the dags folder, the python script inside (test_dontputthescripthere.py) is executed even if the bashoperator isn't telling it to execute. As you can see, the bashoperator is executing the test.py file perfectly, and adds the following line to the csv:

2022-02-21 15:11:53.923284;adding entry with bash python script

The test_dontputthescripthere.py is executed in a loop, and without the bashoperator executing the file. This is all the "- and this is wrong" entries in the demo.csv file.

I suspect some kind of refresh is going on inside airflow, forcing it to execute the python file.

Although it is used in many ETL tasks, Airflow is not the right choice for that kind of operations, it is intended for workflow not dataflow. But there are many ways to do that without passing the whole dataframe between tasks.

You can pass information about the data using xcom.push and xcom.pull:

a. Save the outcome of the first task somewhere (json, csv, etc.)

b. Pass to xcom.push information about saved file. E.g. file name, path.

c. Read this filename using xcom.pull from the other task and perform needed operation.

Or:

Everything above using some database tables:

a. In task_1 you can download data from table_1 in some dataframe, process it and save in another table_2 (df.to_sql()).

b. Pass the name of the table using xcom.push.

c. From the other task get table_2 using xcom.pull and read it with df.read_sql().

Information on how to use xcom you can get from airflow examples. Example: https://github.com/apache/airflow/blob/main/airflow/example_dags/tutorial_etl_dag.py

IMHO there are many other better ways, I have just written what I tried.

The error message tells the problem. ti is not defined.

When you set provide_context=True, Airflow makes Context available for you in the python callable. One of the attributes is ti (see source code). So you need to extract it from kwargs or set it in the function signature.

Your code should be:

Sourav, tell me if this helps:

In an Airflow DAG we generally don't share data between tasks, even though it's technically possible. We're encouraged to keep every task idempotent, not unlike a "pure function" in functional programming. This means that given an input x, a given task will always create the same result.

The DAG you're defining here is basically a blueprint for a data pipeline. When the DAG and tasks are evaluated by the Airflow scheduler, the functions which will be called by the tasks are... well, not yet called. Intuitively, therefore I would expect x to always equal zero, and while it's an interesting mystery to unravel why it isn't always, mutating global variables during a DAG run isn't what Airflow is set up to do.

That said, one simple way to reliably mutate x or c and use it across tasks is to store it in an Airflow variable:

I do not have playground to test it, but I think you should not use f-string for sql parameter. If you use {{something}} in f-string it returns string {something} so parameters for query are not inserted and this results in SQL syntax error as query is run without parameters. Please try to remove f before string for sql in 2nd task.

Your problem is that you did not set dependency between the tasks so inspect_dog may run before or in parallel to get_dog when this happens get_dog will see no xcom value because inspect_dog didn't push it yet.

You just need to set dependency as: