nodejs-bigquery | js client for Google Cloud BigQuery | GCP library
kandi X-RAY | nodejs-bigquery Summary
kandi X-RAY | nodejs-bigquery Summary
Google BigQuery Client Library for Node.js. A comprehensive list of changes in each version may be found in the CHANGELOG. Read more about the client libraries for Cloud APIs, including the older Google APIs Client Libraries, in Client Libraries Explained.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of nodejs-bigquery
nodejs-bigquery Key Features
nodejs-bigquery Examples and Code Snippets
Community Discussions
Trending Discussions on nodejs-bigquery
QUESTION
It seems quite new, but just hoping someone here has been able to use nodejs to write directly to BigQuery storage using @google-cloud/bigquery-storage.
There is an explanation of how the overall backend API works and how to write a collection of rows atomically using BigQuery Write API but no such documentation for nodejs yet. A recent release 2.7.0 documents the addition of said feature but there is no documentation, and the code is not easily understood.
There is an open issue requesting an example but thought I'd try my luck to see if anyone has been able to use this API yet.
...ANSWER
Answered 2021-Nov-19 at 12:50Suppose you have a BigQuery table called student with three columns id,name and age. Following steps will get you to load data into the table with nodejs storage write api.
Define student.proto file as follows
QUESTION
Is there a way using the NodeJS lib for BigQuery (https://github.com/googleapis/nodejs-bigquery) to delete rows? Could not find anything in documentation (https://googleapis.dev/nodejs/bigquery/latest/index.html)
It appears it should be possible using a DELETE statement via DML (https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#delete_examples) but I could not find any examples of how to run this example via the NodeJS lib.
...ANSWER
Answered 2020-Jun-23 at 04:56If you know how to execute a SELECT
query in BigQuery, you do the same to execute a DELETE
query.
QUESTION
TL:DR I'd like to combine the power of BigQuery with my MERN-stack application. Is it better to (a) use nodejs-biquery to write a Node/Express API directly with BigQuery, or (b) create a daily job that writes my (entire) BigQuery DB over to MongoDB, and then use mongoose to write a Node/Express API with MongoDB?
I need to determine the best approach for combining a data ETL workflow that creates a BigQuery database, with a react/node web application. The data ETL uses Airflow to create a workflow that (a) backs up daily data into GCS, (b) writes that data to BigQuery database, and (c) runs a bunch of SQL to create additional tables in BigQuery. It seems to me that my only two options are to:
- Do a daily write/convert/transfer/migrate (whatever the correct verb is) from BigQuery database to MongoDB. I already have a node/express API written using mongoose, connected to a MongoDB cluster, and this approach would allow me to keep that API.
- Use the nodejs-biquery library to create a node API that is directly connected to BigQuery. My app would change from MERN stack (BQ)ERN stack. I would have to re-write the node/express API to work with BigQuery, but I would no longer need the MongoDB (nor have to transfer data daily from BigQuery to Mongo). However, BigQuery can be a very slow database if I am looking for a single entry, a since its not meant to be used as Mongo or a SQL Database (it has no index, one row retrieve query run slow as full table scan). Most of my APIs calls are for very little data from the database.
I am not sure which approach is best. I don't know if having 2 databases for 1 web application is a bad practice. I don't know if it's possible to do (1) with the daily transfers from one db to the other, and I don't know how slow BigQuery will be if I use it directly with my API. I think if it is easy to add (1) to my data engineering workflow, that this is preferred, but again, I am not sure.
...ANSWER
Answered 2020-Apr-29 at 20:25FWIW in a past life, I worked on a web ecommerce site that had 4 different DB back ends. ( Mongo, MySql, Redis, ElasticSearch) so more than 1 is not an issue at all, but you need to consider one as the DB of record, IE if anything does not match between them, one is the sourch of truth, the other is suspect. For my example, Redis and ElasticSearch were nearly ephemeral - Blow them away and they get recreated from the unerlying mysql and mongo sources. Now mySql and Mongo at the same time was a bit odd and that we were dong a slow roll migration. This means various record types were being transitioned from MySql over to mongo. This process looked a bit like: - ORM layer writes to both mysql and mongo, reads still come from MySql. - data is regularly compared. - a few months elapse with no irregularities and writes to MySql are turned off and reads are moved to Mongo.
The end goal was no more MySql, everything was Mongo. I ran down that tangent because it seems like you could do similar - write to both DB's in whatever DB abstraction layer you used ( ORM, DAO, other things I don't keep up to date with etc.) and eventually move the reads as appropriate to wherever they need to go. If you need large batches for writes, you could buffer at that abstraction layer until a threshold of your choosing was reached before sending it.
With all that said, depending on your data complexity, a nightly ETL job would be completely doable as well, but you do run into the extra complexity of managing and monitoring that additional process. Another potential downside is the data is always stale by a day.
QUESTION
On GCP
, I created a CloudFunction
which is trigged by billing events from Pub/Sub
and is publishing some messages on Slack. I am using node.js 10
.I have the following dependencies:
ANSWER
Answered 2020-Mar-21 at 13:40The issue had nothing to do with the asyn ... wait
but was a mistake in the in the way I was preparing the data to write in BigQuery:
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install nodejs-bigquery
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page