dag | : running : Dag is opinionated expression data generator

Support for _PIP_ADDITIONAL_REQUIREMENTS environment variable has not been released yet. It is only supported by the developer/unreleased version of the docker image. It is planned that this feature will be available in Airflow 2.1.1. For more information, see: Adding extra requirements for build and runtime of the PROD image.

For the older version, you should build a new image and set this image in the docker-compose.yaml. To do this, you need to follow a few steps.

1. Create a new Dockerfile with the following content:

Airflow offers using Pools as a way to limit concurrency to an external service.

You can create a Pool via the UI: Menu -> Admin -> Pools

Or with CLI :

You don't need to create your dictionary from the corpus - instead, create a single dictionary entry for your locality list, and look that up to generate a count of each locality. You can then count them by compiling the dfm, and then converting the feature of that dictionary key into a logical to get the vector you want.

You could use the substitute family of bash functions (see Nixpkgs manual) or use the substituteAll Nix function which produces a derivation that performs the substitution.

You are probably running it on MacOS. Set:

The command parameter of SSHOperator is templated thus you can get the xcom directly:

The module fractions is part of the Python standard library. There used to be a function gcd, which, as the linked documentation says, is:

Deprecated since version 3.5: Use math.gcd() instead.

Since the function gcd was removed from the module fractions in Python 3.9, it seems that the question uses Python 3.9, not Python 3.7.6 as the question notes, because that Python version still had fractions.gcd.

The error is raised by networkx. Upgrading to the latest version of networkx is expected to avoid this issue:

The TriggerDagRunOperator now takes a conf parameter to which a dictinoary can be provided as the conf object for the DagRun. Here is more information on triggering DAGs which you may find helpful as well.

EDIT

Since you need to execute a function to determine which DAG to trigger and do not want to create a custom TriggerDagRunOperator, you could execute intakeFile() in a PythonOperator (or use the @task decorator with the Task Flow API) and use the return value as the conf argument in the TriggerDagRunOperator. As part of Airflow 2.0, return values are automatically pushed to XCom within many operators; the PythonOperator included.

Here is the general idea:

I think the Snakefile is ok, SRR3350597_GSM2112330_RA_hip_3_Homo_sapiens_RNA-Seq_1.fastq.gz is simply missing. See the ls output of yours, that file is not in it.