pandas-gbq | Pandas Google BigQuery | GCP library

I have two Azure accounts. And I tried to deploy the same function to these two accounts (to the function apps). The deployment to the 1st account - successful, but to the 2nd account - failed.

The only big difference between the two accounts is that I do not have direct access to the resource group that the 2nd account's function app uses (I have access to the resource group at the 1st account). May it be the reason why I can't deploy the program to the function app at the 2nd account?

Deploy output of the function app at the 1st account:

I have a BigQuery table that I am hoping to populate using pandas-gbq. The table has a predefined schema that includes nullable int and string fields. Currently, I am generating a dict of one list for each data field and putting pandas.NA or None (I've tried both) when I am missing values. I am currently missing values for one of my nullable int fields, e.g.:

data source: https://catalog.data.gov/dataset/nyc-transit-subway-entrance-and-exit-data

I tried looking for a similar problem but I can't find an answer and the error does not help much. I'm kinda frustrated at this point. Thanks for the help. I'm calculating the closest distance from a point.

I build a Docker image for an armv7 architecture with python packages numpy, scipy, pandas and google-cloud-bigquery using packages from piwheels. The base image is Python:3.7-buster.

If I'm running a container with this image, the container always restarts and gives me the error log "ValueError: This method requires pyarrow to be installed":

To speed up my cluster instantiation time, I've created a custom image with all the additional dependencies installed using miniconda3 available for dataproc image 1.5.34-debian10. (I followed the steps here: GCP Dataproc custom image Python environment to ensure I used the correct python environment).

However, when I start my cluster with --optional-components ANACONDA,JUPYTER my custom dependencies are removed and I'm left with a base installation of anaconda and jupyter. I assume the anaconda installation is overwriting my custom dependencies. Is there any way to ensure my dependencies aren't overwritten? If not, is it possible to install anaconda and jupyter as part of my custom dataproc image instead?

I've used the following command to create the custom image:

I am currently using Airflow 1.10.9 on ECS. I explicitly specify Airflow version in my requirements.txt file as below.

• OS type and version: 1.5.29-debian10
• Python version: 3.7
• google-cloud-bigquery version: 2.8.0

I'm provisioning a dataproc cluster which gets the data from BigQuery into a pandas dataframe. As my data is growing I was looking to boost the performance and heard about using the BigQuery storage client.

I had the same problem in the past and this was solved by setting the google-cloud-bigquery to version 1.26.1. If I use that version I get the following message.

I'm facing a very strange error from few days now. I have a python2.7 project that was running smoothly but since few days its been throwing an error:

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

I'm using virtual environment for my project. What happened was that few days ago I tried installing nginx using brew command and what I believe is brew updated some dependencies that were being used for python2.7 project (this is what i think might be the case). Now since that day, I'm facing this issue and I have googled it everywhere but couldn't resolve. Below is some information you might need to figure out.

my requirements.txt file

As per the Python BigQuery client documentation, it seems that multiprocessing should work. But I keep getting an error when trying a simple load to a BigQuery table from a pandas dataframe using multiprocessing and I wonder if the following statement from the doc would have anything to do with it.

In multiprocessing scenarios, the best practice is to create client instances after multiprocessing.Pool or multiprocessing.Process invokes os.fork().

I wrote my code based on this GCP doc (google-cloud-bigquery), that just tries to create 2 processes to load two different pandas dataframe on the same table (I have also tried to load them on two different tables and got the same error):