pubsub-to-bigquery | highly configurable Google Cloud Dataflow pipeline | GCP library
kandi X-RAY | pubsub-to-bigquery Summary
kandi X-RAY | pubsub-to-bigquery Summary
A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Entry point for testing
- Parses a XML document from the provided parameters
pubsub-to-bigquery Key Features
pubsub-to-bigquery Examples and Code Snippets
Community Discussions
Trending Discussions on pubsub-to-bigquery
QUESTION
Some jobs are remaining with pending pending state and I can't cancel them.
How do I cancel the job.
Web console shows like this.
- "The graph is still being analyzed."
- All logs are "No entries found matching current filter."
- Job status: "Starting..." There isn't appered a cancel button yet.
There are no instances in the Compute Engline tab.
What I did is below. I created a streaming job. it was simple template job, Pubsub subscription to BigQuery. I set machineType as e2-micro because it was just a testing.
I also tried to drain and cancel by gcloud but it doesn't work.
...ANSWER
Answered 2021-Feb-10 at 12:47QUESTION
I have set up a GCloud Dataflow pipeline which consumes messages from a Pub/Sub subscription, converts them to table rows and writes those rows to a corresponding BigQuery table.
Table destination is decided based on the contents of the Pub/Sub message and will occasionally lead to the situation that a table does not exist yet and has to be created first. For this I use create disposition CREATE_IF_NEEDED
, which works great.
However, I have noticed that if I manually delete a newly created table in BigQuery while the Dataflow job is still running, Dataflow will get stuck and will not recreate the table. Instead I get an error:
...ANSWER
Answered 2020-Mar-22 at 14:36This is not possible in the current BigqueryIO
connector. From the github link of the connector present here you will observe that for StreamingWriteFn
which is what your code, the table creation process is done in getOrCreateTable
and this is called in finishBundle
. There is a map of createdTables
that is maintained and in finishBundle
the table gets created if it not is already present, once it is present and stored in the hashmap it is not re-created as shown below:-
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install pubsub-to-bigquery
You can use pubsub-to-bigquery like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the pubsub-to-bigquery component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page