table_builder | Rails builder for creating tables | Calendar library

If you're trying to install feather (the package for "Feather: fast, interoperable binary data frame storage" then you should execute pip install feather-format. The PyPI feather package is different and appears to only be for Python 2.x.

Try move db object into models.py. I omitted some unnecessary code.

__init__.py

I fought this use case for a long time. In short, a dag that’s built based on the state of a changing resource, especially a db table, doesn’t fly so well in airflow.

My solution was to write a small custom operator that’s a subclass if truggerdagoperator, it does the query and then triggers dagruns for each of the subprocess.

It makes the process “join” downstream more interesting, but in my use case I was able to work around it with another dag that polls and short circuits if all the sub processes for a given day have completed. In other cases partition sensors can do the trick.

I have several use cases like this (iterative dag trigger based on a dynamic source), and after a lot of fighting with making dynamic Subdags work (a lot), I switched to this “trigger subprocess” strategy and have been doing well since.

Note - this may make a large number of dagruns for one targ (the target). This makes the UI challenging in some places, but it’s workable (and I’ve started querying the db directly because I’m not ready to write a plugin that does UI stuffs)

It's less of an issue about OOP with airflow and more about state with airflow.

Any state that needs to be passed between tasks needs to be stored persistently. This is because each airflow task is an independent process (which could even be running on a different machine!) and thus in-memory communication is not possible.

You are correct you can use XCOM to pass this state (if it's small, since it gets stored in the airflow database). If it's large you probably want to store it somewhere else, maybe a filesystem or S3 or HDFS or a specialized database.

It looks like you also need to be linking snappy. I assume this is because leveldb is also using that library. So you'll want to download that an install it as either a static or shared library and link it using -lsnappy. Make sure you place the linker flag to snappy after the linker flag for leveldb so it can fill in the symbols that leveldb is missing.