s3hook | Transparent Client-side S3 Request | HTTP Client library

I'm creating the below class that is based off the s3CopyObjectOperator, but I have to copy all the files from an s3 directory and save to another directory, then delete the files.

But I need the file names from the directory I'm copying from. So lets say the Copy Source is:

I need to implement the html_content dynamic for custom email operator, as we have html_content different for different jobs.

Also, I need the values, for example, rows and filename be dynamic

The example below is one of the email body:

I'm using S3Hook in my task to download files from s3 bucket on DigitalOcean spaces. Here is an example of credentials which are perfectry working with boto3, but causing errors when used in S3Hook:

I'm learning Airflow and I'm trying to understand how connections works.

I have a first dag with the following code:

I am trying to move s3 files from a "non-deleting" bucket (meaning I can't delete the files) to GCS using airflow. I cannot be guaranteed that new files will be there everyday, but I must check for new files everyday.

my problem is the dynamic creation of subdags. If there ARE files, I need subdags. If there are NOT files, I don't need subdags. My problem is the upstream/downstream settings. In my code, it does detect files, but does not kick off the subdags as they are supposed to. I'm missing something.

here's my code:

I've read the documentation for creating an Airflow Connection via an environment variable and am using Airflow v1.10.6 with Python3.5 on Debian9.

The linked documentation above shows an example S3 connection of s3://accesskey:secretkey@S3 From that, I defined the following environment variable:

AIRFLOW_CONN_AWS_S3=s3://#MY_ACCESS_KEY#:#MY_SECRET_ACCESS_KEY#@S3

And the following function

I am using docker-compose to set up a scalable airflow cluster. I based my approach off of this Dockerfile https://hub.docker.com/r/puckel/docker-airflow/

My problem is getting the logs set up to write/read from s3. When a dag has completed I get an error like this

I am using the Airflow EMR Operators to create an AWS EMR Cluster that runs a Jar file contained in S3 and then writes the output back to S3. It seems to be able to run the job using the Jar file from S3, but I cannot get it to write the output to S3. I am able to get it to write the output to S3 when running it as an AWS EMR CLI Bash command, but I need to do it using the Airflow EMR Operators. I have the S3 output directory set both in the Airflow step config and in the environment config in the Jar file and still cannot get the Operators to write to it.

Here is the code I have for my Airflow DAG

I'm still in the process of deploying Airflow and I've already felt the need to merge operators together. The most common use-case would be coupling an operator and the corresponding sensor. For instance, one might want to chain together the EmrStepOperator and EmrStepSensor.

I'm creating my DAGs programmatically, and the biggest one of those contains 150+ (identical) branches, each performing the same series of operations on different bits of data (tables). Therefore clubbing together tasks that make-up a single logical step in my DAG would be of great help.

Here are 2 contending examples from my project to give motivation for my argument.

1. Deleting data from S3 path and then writing new data

This step comprises 2 operators

• DeleteS3PathOperator: Extends from BaseOperator & uses S3Hook
• HadoopDistcpOperator: Extends from SSHOperator

2. Conditionally performing MSCK REPAIR on Hive table

This step contains 4 operators

• BranchPythonOperator: Checks whether Hive table is partitioned
• MsckRepairOperator: Extends from HiveOperator and performs MSCK REPAIR on (partioned) table
• Dummy(Branch)Operator: Makes up alternate branching path to MsckRepairOperator (for non-partitioned tables)
• Dummy(Join)Operator: Makes up the join step for both branches

Using operators in isolation certainly offers smaller modules and more fine-grained logging / debugging, but in large DAGs, reducing the clutter might be desirable. From my current understanding there are 2 ways to chain operators together

1. Hooks

   Write actual processing logic in hooks and then use as many hooks as you want within a single operator (Certainly the better way in my opinion)

2. SubDagOperator

   A risky and controversial way of doing things; additionally the naming convention for SubDagOperator makes me frown.

My questions are

• Should operators be composed at all or is it better to have discrete steps?
• Any pitfalls, improvements in above approaches?
• Any other ways to combine operators together?
• In taxonomy of Airflow, is the primary motive of Hooks same as above, or do they serve some other purposes too?

UPDATE-1

3. Multiple Inhteritance

While this is a Python feature rather than Airflow specific, its worthwhile to point out that multiple inheritance can come handy in combining functionalities of operators. QuboleCheckOperator, for instance, is already written using that. However in the past, I've tried this thing to fuse EmrCreateJobFlowOperator and EmrJobFlowSensor, but at the time I had run into issues with @apply_defaults decorator and had abandoned the idea.