bigquery | Golang BigQuery API Wrapper | REST library

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

1. For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.

2. For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

You can do the following:

Here you have a working example:

There is no LAST_INDEX method you can you for the pagination, as you can check in the documentation.

About your request:

I want to read data in chunks like the first 10 records in one process, then the next 20 records in another process and etc.,

Using python you can use some parameters as max_results and start_index to perform it, but in java the only way will be paginating on your query, and change it for each process. So for each process in parallel you will have a different query.

So, each process will have to:

1. Order by some field (or by all the fields) to guarantee every query will return the data in the same order
2. Paginate using limit and offset:

i.e:

I've locally tested building a container image using Cloud Build Client Python library. It turns out to have the same error even the Dockerfile file is existing in current directory:

error:

Step #0: unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /workspace/Dockerfile: no such file or directory

build steps:

The issue seems to be there for both Cloud Spanner and BigQuery. Tried with different values but the issue seems to be for a particular set of inputs i.e. it is showing the unexpected result for values 33.092136, 34.092136, 35.092136, ........., 62.092136, 63.092136. Before 33.092136 and from 64.092136 onwards the issue seems to be not there. Also I tried with Cloud SQL(MySQL) and the issue is not there.

I have created an issue in Public Issue Tracker for the same. I would suggest you star the issue so that you will get notified whenever there is any update on the created issue.

If a table is truncated while the streaming pipeline is still going on or performing a streaming insertion on a recently truncated table, you could receive some errors like mentioned in the question (Table is truncated), that's expected behavior. The metadata consistency mode for the InsertAll (very high QPS API) is eventually consistent, this means that when using the InsertAll API, it may get delayed table metadata and returns the failure like table truncated. The typical way to resolve this issue is to back-off and retry.

Currently, there is no option in the BigQuery API to check if the table is in truncated state or not.

Suppose you have a BigQuery table called student with three columns id,name and age. Following steps will get you to load data into the table with nodejs storage write api.

Define student.proto file as follows

I followed google documentation https://cloud.google.com/bigquery/docs/materialized-views to create materialized view for your requirement.

I inserted the data in a table as below

I ran below query to create a materialized view