ETL-pipeline | Educational project on how to build an ETL ( Extract | Data Migration library

The short answer is that this is exactly how you define retries. When you get Airflow up and running with its scheduler component, this will work exactly as you expect.

Airflow essentially builds models that define how to execute compute tasks, but in production it uses the scheduler to add and evaluate data to make sure those compute tasks are run at the right time. As a consequence, some of the models and features work specifically for pieces inside its own architecture.

First, Airflow has its CLI which will allow you to use a command to run a specified compute task. This is the result of your command. It essentially finds the right compute task, generates a task instance with the date value you pass to it and executes it.

However, on top of the CLI, Airflow has a system component called a scheduler. The scheduler uses the underlying data models to essentially calculate and determine which compute task has met its dependencies and requires a run.

In that context, the scheduler repeatedly scans your DAGs to see if there are Tasks that have all of their dependencies met. If they have, it generates a Task Instance with retries=0. An executor performs the compute. The executor takes the information from trying to execute the task and updates the model or backend database. If it failed it also gives the status field a failed status. The next time the scheduler reads the database it will see a failed task, it will then compare the retries value against the configured retries. If it deduces that it is allow to retry again retries < max_retries-1 and the state is failed, it will change its state, add 1 to retries, and pass the instruction to the executor.

Alternatively, if retries == max_retries - 1 the scheduler will set the status of the Task to failed, notify that the task has failed, and not retry the task.