airflow-docker | Apache Airflow Docker Image | Continuous Deployment library

LETS GOOOOOOOOOO!
PAN COMIDO!
DU GATEAU!
Finally got it to work :). So the main issue was the fact that I didn't have all the required packages. So I tried doing just pip install configparser in the container and this actually helped for one of the DAGs I had to run. However this didn't seem sustainable nor practical so I decided to just go ahead with the Dockerfile method in effect extending the image. I believe this was the way they called it. So here's my Dockerfile \

I'm launching the whole setup from a docker-compose.yml where I set AIRFLOW_UID=1234 and AIRFLOW_GID=0. I'm using a docker image based on the official airflow image with the addition that I have created 'newuser' with gid=1234 and 'docker' group with gid that matches the one at the host.

You should not do it at all. The user will be created automatically by Airflow's image entrypoint when you use a differnt UID than default - see https://airflow.apache.org/docs/docker-stack/entrypoint.html#allowing-arbitrary-user-to-run-the-container. In fact all that you want to do should be possible without having to extend the Airflow image.

What you need to do, you need to create this user that you want to run inside the container ON THE HOST - not in the container. And it should belong to the docker group ON THE HOST - not in the container.

Docker works in the way that it uses the same kernel/users that are defined in the system, so when you run something as a user in the container, it is run with the "host" user priviledges, so you you map your docker socket to within the container, it will be able to use the socket/run docker command becaue it will have the right permissions on the host.

Therefore (in case you run your docker-compose as regular user who already belongs to docker group) the best way is the one suggested in the quick-start - i.e. run airflow with your "host" user that you are logged in with: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

This also makes all the files created in container belong to the "logged in user" (if they are created in directories mounted inside - such as logs directory).

But if your goal is to use it in "unattended" environment, then likely creating the new user on your host and adding the user to both 0 and docker groups should solve the problem.

There is a pretty detailed guide on how to achieve what you are looking for on the Airflow docs here. Depending on your requirements, this may be as easy as extending the original image using a From directive while creating a new Dockerfile, or you may need to customize the image to suit your needs.

If you go with the Extending the image approach your new Dockerfile will be something like this:

If you want the port 8080 to be exposed to the host you could use -p parameter in docker run commannd. Also you can set the command webserver directlry while starting the container. This will start Airflow with Sequential Executor.

I am pretty certain this issue is because the s3 logging configuration has not been set on the worker pods. The worker pods don't get given configuration set using environment variables such as AIRFLOW__CORE__REMOTE_LOGGING: True. If you wish to set this variable in the worker pod then you must copy the variable and append AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__ to the copied environment variable name: AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__REMOTE_LOGGING: True.

In this case you would need to duplicate all of your variables specifying config for s3 logging and append AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__ to the copies.