python-daemon | Python daemonizer for Unix , Linux and OS X

Your database name is "postgres", you can see on psql bash (postgres=#), but you setup it to "dbname" on settings

It seems that you are confusing the use-cases for PythonVirtualenvOperator and PythonOperator.

If you simply want to run a Python callable in a task (callable_virtualenv() in your case) you can use PythonOperator. In this case, it does not matter if you installed Airflow in a virtual environment, system wide, or using Docker.

What happens in your code is the following: PythonVirtualenvOperator creates another virtual environment (which is completely unrelated to the one in which you run Airflow), installs Airflow into it, and tries to import Variable. But this another Airflow installation is not configured and that is why you get those exceptions. You could set the AIRFLOW_HOME environment variable for this second Airflow installation to the same directory as used by the first Airflow installation, and this should actually work, but it looks like an overkill to me.

So, what you can do is install colorama into the same environment in which you installed Airflow and replace PythonVirtualenvOperator by PythonOperator.

BTW, those print() inside the callable would be redirected into a log file and not printed to terminal, so it probably does not make much sense to use colorama with them.

I am on MacOS 10.15.7 Python version 3.8.2

I'm guessing you used the Python 3 bundled/pre-installed with macOS Catalina.

The documentation gives what you want : pip download

pip download does the same resolution and downloading as pip install, but instead of installing the dependencies, it collects the downloaded distributions into the directory provided

source

So you may try these option with pip download :

I was able to figure out a work around that utilizes Threading instead of a daemon. See below for the code.

Launch Daemons (and cron jobs and ...) don't run under your usual Terminal/shell environment, so they don't get whatever customization, add-ons, etc are set up in your Terminal/shell environment. In particular, they don't run the various shell initialization scripts (~/.bash_profile, ~/.bashrc, etc) that normally have the commands to set up your environment.

In the specific case of anaconda, its installer adds a section to your ~/.bashrc file to add the anaconda binaries directory to PATH, and probably a bunch of other changes. This affects your environment your bash sessions in Terminal, but doesn't take effect for Launch Agents, or even Terminal sessions in other shells (zsh is now the default, and it's causing some trouble with this).

For a Launch Daemon, I'd recommend against making it run your shell setup scripts -- those are your personal config, and don't have any real business being run by a system process. Instead, I'd create a short shell script that includes the relevant setup (copied from your ~/.bashrc), and then runs your python script. Then change the Launch Daemon so it runs that script, rather than running the python script directly.

P.S. Since the script doesn't have to do anything after starting the python script, this is a case where the script could use the exec command to exit itself, and run the python script instead in the same process. That means that python would be a direct subprocess of launchd, and launchd can monitor, control, etc it in the way that it likes to. (That's as opposed to having the script run python normally, in which case you'd have a now-pointless shell process hanging out waiting for the python script to finish so that it can finish.)

So I'd end the script with:

Your issue is typically related to Xcode toolkit.

Type this command in your terminal

The error "Failed to install PyPI packages" means Cloud Build was unable to build Docker images for your Cloud Composer environment. This is usually because there was a PyPI package installation failure, or there were incompatible dependencies. In your case, it is the latter: