FileSensor | Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具 | Crawler library

Yhis should do the job:

I am not sure if current Airflow approach is best for this use case actually. In the current incarnation Airflow is really all about working on "data intervals" - so basically each "dag run" is connected to some "data interval" and it should be processing data for that data interval. Classic Batch processing.

If I understand your case is more like a streaming (not entirely) but close. You get some (subset of) data which arrived since the last time and you process that data. This is not what (again current) version of Airflow - not even 2.1 is supposed to handle - because there is a complex manipulation of "state" which is not "data interval" related (and Airflow currently excels in the "data interval" case).

You can indeed do some custom operators to handle that. I think there is no ready-to-reuse pattern in Airflow for what you want to achieve, but Airflow is flexible enough that if you write your own operators you can certainly work around it and implement what you want. And writing operators in Airflow is super easy - it's a simple Python class with "execute" which can reuse existing Hooks to reach out to external services/storages and use XCom for communication between tasks. It's surprisingly easy to add a new operator doing even complex logic (and again - reusing hooks to make it easier to communicate with external services). For that, I think it's still worth to use Airflow for what you want to do.

How I would approach it - rather than modifying the timestamps of the files, I'd create other files - markers - with the same names, different extensions and base my logic of processing on that (this way you can use the external storage as the "state"). I think there will be no ready "operators" or "sensors" to help with it, but again - writing custom one is easy and should work.

However soon (several months) in Airflow 2.2 (and even more in 2.3) we are going to have some changes (mainly around flexible scheduling and decoupling dag runs from data intervals and finally to allow dynamic DAG with flaxible structure that can change per-run) that will provide some nice way of handling cases similar to yours.

Stay tuned - and for now rely on your own logic, but look out for simplifying that in the future when Airflow will be better suited for your case.

And in the meantime - do upgrade to Airflow 2. It's well worth it and Airflow 1.10 reached end of life in June, so the sooner you do, the better - as there will not be any more fixes to Airflow 1.10 (even critical security fixes)

Support for _PIP_ADDITIONAL_REQUIREMENTS environment variable has not been released yet. It is only supported by the developer/unreleased version of the docker image. It is planned that this feature will be available in Airflow 2.1.1. For more information, see: Adding extra requirements for build and runtime of the PROD image.

For the older version, you should build a new image and set this image in the docker-compose.yaml. To do this, you need to follow a few steps.

1. Create a new Dockerfile with the following content:

I wasn't able to reproduce the issue. The FileSensor is waiting for file in directory which can be specified in connection used by the operator. By default it is ~/ directory. Are you sure that the file is created in place where you expect it to be? Currently you sensor is waiting for /testfile.csv is that expected?

I created a dummy file and run you DAG and it run ok:

In general I think Elad's suggestion might work, however I would argue it's a bad practice. DAGs are by design (and name) acyclic, so creating any types of loops within it might cause it to behave unexpectedly.

Also based on Airflow documentation you should set your dag schedule to None if you plan to use external dag trigger. Personally I'm not sure if it will necessarily break something, but it definitely can give you outputs you don't expect. Probably will take you longer to debug it later as well, if something goes wrong.

IMHO better approach would be for you try and to rethink your design. In case you need to reschedule dag on failure you can take advantage of reschedule mode for sensor https://www.astronomer.io/guides/what-is-a-sensor . Not sure why you would want to re-run it on success, if it's the case of multiple files in the source, I would say rather create multiple sensors with variable parameter and for loop in your dag script.

In short: if the task writes where the DagBag reads from, yes, but it's best to avoid a pattern that requires this. Any DAG you're tempted to custom-create in a task should probably instead be a static, heavily parametrized, conditionally-triggered DAG. y2k-shubham provides an excellent example of such a setup, and I'm grateful for his guidance in the comments on this question.

That said, here are the approaches that would accomplish what the question is asking, no matter how bad of an idea it is, in the increasing degree of ham-handedness:

• If you dynamically generate DAGs from a Variable (like so), modify the Variable.
• If you dynamically generate DAGs from a list of config files, add a new config file to wherever you're pulling config files from, so that a new DAG gets generated on the next DAG collection.
• Use something like Jinja templating to write a new Python file in the dags/ folder.

To retain access to the task after it runs, you'd have to keep the new DAG definition stable and accessible on future dashboard updates / DagBag collection. Otherwise, the Airflow dashboard won't be able to render much about it.

While it isn't clear, i'm assuming that downstream DAG(s) that you trigger via your orchestrator DAG are NOT dynamically generated for each file (like your Move & TriggerDAG tasks); in other words, unlike your Move tasks that keep appearing and disappearing (based on files), the downstream DAGs are static and stay there always

You've already built a relatively complex workflow that does advanced stuff like generating tasks dynamically and triggering external DAGs. I think with slight modification to your DAGs structure, you can get rid of your troubles (which also are quite advanced IMO)

1. Relocate the Move task(s) from your upstream orchestrator DAG to the downstream (per-file) process DAG(s)
2. Make the upstream orchestrator DAG do two things
3. Sense / wait for files to appear
4. For each file, trigger the downstream processing DAG (which in effect you are already doing).

For the orchestrator DAG, you can do it either ways

1. have a single task that does file sensing + triggering downstream DAGs for each file
2. have two tasks (I'd prefer this)
   • first task senses files and when they appear, publishes their list in an XCOM
   • second task reads that XCOM and foreach file, triggers it's corresponding DAG

but whatever way you choose, you'll have to replicate the relevant bits of code from

• FileSensor (to be able to sense file and then publish their names in XCOM) and
• TriggerDagRunOperator (so as to be able to trigger multiple DAGs with single task)

here's a diagram depicting the two tasks approach

Seems like you should be able to run a batch processor dag with a bash operator to clear the folder, just make sure you set depends_on_past=True on your dag to make sure the folder is successfully cleared before the next time the dag is scheduled.

If you wanted to surely run either both scripts or none I would add a dummy task before the two tasks that need to run in parallel. Airflow will always choose one branch to execute when you use the BranchPythonOperator.

I would make these changes: