docker-airflow | building docker based airflow image | Continuous Deployment library

Don't use Puckel Docker Image. It's not been maintained for years, Airflow 1.10 has reached End Of Life in June 2021. You should only look at Airflow 2 and Airflow has official reference image that you can use:

Airflow 2 has also Quick-Start guides you can use - based on the image and docker compose: https://airflow.apache.org/docs/apache-airflow/stable/start/index.html

And it also has Helm Chart that can be used to productionize your setup. https://airflow.apache.org/docs/helm-chart/stable/index.html

Don't waste your (and other's) time on Puckel and Airflow 1.10.

If you add an environment variable named AIRFLOW_VAR_CONFIG_BUCKET to the list under environment:, it should be accessible by Airflow. Sounds like you're doing that correctly.

Two things to note:

• Variables (& connections) set via environment variables are not visible in the Airflow UI. You can test if they exist by executing Variable.get("config_bucket") in code.
• The Airflow scheduler/worker (depending on Airflow executor) require access to the variable while running a task. Adding a variable to the webserver is not required.

The restart and deploy.restart_policy options configure the same thing but depend on the way you run your containers:

• restart is used by Docker Compose
• deploy.restart_policy is used by Docker Swarm

The deploy option is used for Docker Swarm only and is ignored by Docker Compose.

From the documentation on deploy.restart_policy:

Configures if and how to restart containers when they exit. Replaces restart.

And here about restart:

The restart option is ignored when deploying a stack in swarm mode.

You need to run the following to map a volume to share data between the container and the host

see answer here copy file from docker to host system using python script

I recently added a quick start guides to the official Apache Airflow documentation. Unfortunately, this guide has not been released yet. It will be released in Airflow 2.0.1. For now, you can use the development version, and when a stable version is released it will be very easy for you to migrate. I don't expect any major changes to our docker-compose.yaml files. http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/start/docker.html

Within the container, you have to point to the config file in the mounted volume, so change config_parser.read("C:/dags/file.ini") to config_parser.read("/usr/local/airflow/dags/file.ini").

Executable files, i.e. binary files, are dependent on the computer's architecture (amd64, arm...). Docker's image contains binary files. That is, the docker image is computer architecture dependent.

Therefore, if you look at the registry of docker, the OS and Architecture of the image are specified. Refer to the dockerhub/puckel/docker-airflow you used, linux/amd64 You can see it only supports. In other words, it doesn't work in arm architecture. If you want to run this arm architecture there will be several ways, but the point is one. It is to build the result of building the origin code with arm, not amd64, as docker image.

In github.com/puckel/docker-airflow, guidelines for building are well stated.

First, if you look at the Dockerfile provided by the github, it starts from the image FROM python:3.7-slim-buster. For the corresponding python:3.7-slim-buster, it supports linux/arm/v5, linux/arm/v7, linux/arm/v5, linux/arm64/v8. dockerhub/python/3.7-slim-buster

In other words, you can build to arm architecture

I have experience creating images for multiple architectures through the docker buildx command. Of course, other methods exist, but I will only briefly introduce the commands below.

dockerhub/buildx

• docker buildx is an experimental feature, and it is still recommended Experimental features must not be used in production environments.

Since you are using Airflow 1.10.14 The import should be

I think that you're getting Permission denied because you are running docker with user airflow.

Can you try to run it as root? (this is risky! don't use in production - it is just an effort to temporary workaround). Avoid using root user!