gcsfs | Google Cloud Storage filesystem for PyFilesystem2 | Cloud Storage library

Pyarrow's dataset module reads CSV files in chunks (the default is 1MB I think) and it processes those chunks in parallel. This makes column inference a bit tricky and it handles this by using the first chunk to infer data types. So the error you are getting is very common when the first chunk of the file has a column that looks integral but in future chunks the column has decimal values.

If you know the column names in advance then you can specify the data types of the columns:

The release of setuptools 60.3.0 caused AttributeError because of a bug and now Setuptools 60.3.1 is available. You can refer to the GitHub Link here.

For more information you can refer to the stackoverflow answer as :

If you run into this pip error in a Cloud Function, you might consider updating pip in the "requirements.txt" but if you are in such an unstable Cloud Function the better workaround seems to be to create a new Cloud Function and copy everything in there.

The pip error probably just shows that the source script, in this case the requirements.txt, cannot be run since the source code is not fully embedded anymore or has lost some embedding in the Google Storage. or you give that Cloud Function a second chance and edit, go to Source tab, click on Dropdown Source code to choose Inline Editor and add main.py and requirements.txt manually (Runtime: Python).”

If the database is on a VM, and in your VPC, you can create a VPC connector and attach it to your Cloud Function to access it.

If it's deployed else where,

• Either the database has a public IP, and Cloud Functions can directly access it.
• Or the database has a private IP and you need to create a VPN between your VPC and the private foreign network with your database. And again add a serverless VPC connector to Cloud Functions to allow it to your your VPC and the VPN to access the database.

json_normalize expects a list of dicts, not a single nested dict if it is to create multiple rows. You therefore need to 'unpack' your nested dict into a list of dicts, for example by taking the values() of config_queries:

Not sure if I understand this correctly but you seem to be nesting your PythonOperator creation inside the make_csv dependency which is an infinite loop as far as I can see. Maybe try removing that outside of the function and see what happens?

This change is related to the Python 3.7 buildpacks rollout. As a result of the move to gunicorn and its worker model, the global scope and function scope can be executed in separate processes. This issue can be fixed by moving the GCSFileSystem initialization into the function body.

You need to put fs = gcsfs.GCSFileSystem(project='project-name-1234') inside the entrypoint try_gcsfs. Your code should look like this:

I couldn't find a way to update the weights of a pickle file and eventually settled with increasing the timeout parameter in cloud run to more than the training time and it fixed the issue for the time being.

According to the exeption and pandas version, the problem could be that you have non-Unicode character(s) in your file, that was suppressed before v1.3. See this bug report comment.

Also, pandas documentation introduced the encoding_errors parameter (encoding_errors str, optional, default “strict”) in version 1.3 to explicitly handle encoding errors. So you should check your file for incorrect characters.

In any case, if you want the behavior prior v1.3, you can use replace (or ignore if it better for your case):

So simple answer to this appears to be that the file literally has to be saved as "model.pkl". I assumed that the name before the extension could vary but no.

I am still struggling to make a prediction be generated but it now returns the failure within 15 minutes or so