cloud-pubsub | Google Cloud PubSub client in rust | GCP library

The customize_conda.sh script is the recommended way of customizing Conda env for custom images.

If you need more than the script does, you can read the code and create your own script, but anyway you want to use the absolute path e.g., /opt/conda/anaconda/bin/conda, /opt/conda/anaconda/bin/pip, /opt/conda/miniconda3/bin/conda, /opt/conda/miniconda3/bin/pip to install/uninstall packages for the Anaconda/Miniconda env.

It's a bug (harmless) in definition of the google provider 2.2.0 in fact:

In provider.yaml:

airflow.providers.google.common.hooks.leveldb.LevelDBHook

should be:

airflow.providers.google.leveldb.hooks.LevelDBHook

This was fixed in https://github.com/apache/airflow/pull/15453 and will be available in next version of google provider.

In general, such exception happens when you have two versions of the same class in the classpath. Some of the reasons that may happens are:

1. Your dependencies include two version of google-api-client. I don't see any direct dependency so It could be a transitive dependency. You can run mvn dependency:tree and look for google-api-client dependencies and exclude one of the jars
2. You are packing a provided dependency. It could be that app engine already include such jar with HttpTransport, if you also include this HttpTransport class into your artefact it will create problems. To fix this identify which dependency contains HttpTransport class and in your pom.xml add provided to it
3. This class is included in the classpath from some old jar in some generated repository. If this is the case all you have to do is reset your environment. Clone the repository again and re-init app engine

The error message shows you the failing dependencies. The group IDs are all wrong and you have a typo on HikariCP. Here are the correct declarations:

This will take very long to run, but you could do the following steps:

1. Add a step to your workflow to install pyenv. You will also need to make sure the shim is available to your PATH.
2. Add a step to your workflow to pyenv install 3.6.5
3. Then checkout your repo.
4. Add a step to set the local Python version to 3.6.5: pyenv local 3.6.5.
5. Run pytest.

I anticipate that steps 1 and 2 will take the longest. You can speed this up by perma-caching pyenv and Python 3.6.5 if you know where the files are stored. I've asked and answered myself on how to perma-cache a tool between workflow runs here.

The issue is on the dependency itself rather than Cloud Functions as I was able to replicate this problem on my machine with Python 3.8 and Pandas 1.2.0 installed.

To remove the empty row, remove "lines" on dataframe.to_json:

In order to further contribute to the community I am posting the answer based on my comment above.

Firstly, you should try to upgrade the packages using the command:

pip install --upgrade pandas-gbq 'google-cloud-bigquery[bqstorage,pandas]'

Then, instead of using the to_dataframe() method you can use the read_gbq(), which loads data from BigQuery using the environment's default project, as follows:

First of all - I have a really bad experience with php and pubsub because of the php PubSubClient. If your script is only waiting for push and checking the messages then remove the pubsub package and handle it with few lines of code.

Example:

We have encountered a similar issue. Probably the GCP internal load balancer in front of your cloud run can't pass the request to the instance. This means that some processes made the cloud run instance stall after 60 seconds, so that it does not receive any request. According to this post, it might have something to do with cloud run interfering with the gunicorn workers. Since cloud run (managed) is a serverless environment, the order in which workers and code are loaded and shut down matters. You could try setting --preload and --timeout=0. Another article suggests a similar thing.