pubsub-to-bigquery | highly configurable Google Cloud Dataflow pipeline | GCP library

by bomboradata Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | pubsub-to-bigquery Summary

pubsub-to-bigquery is a Java library typically used in Cloud, GCP applications. pubsub-to-bigquery has build file available, it has a Permissive License and it has low support. However pubsub-to-bigquery has 1 bugs and it has 1 vulnerabilities. You can download it from GitHub.

A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub

Support

Quality

Security

License

Reuse

kandi-support Support

pubsub-to-bigquery has a low active ecosystem.
It has 64 star(s) with 6 fork(s). There are 7 watchers for this library.
It had no major release in the last 12 months.
There are 0 open issues and 1 have been closed. On average issues are closed in 435 days. There are no pull requests.
It has a neutral sentiment in the developer community.
The latest version of pubsub-to-bigquery is current.

pubsub-to-bigquery Support

Best in #GCP

Average in #GCP

pubsub-to-bigquery Support

Best in #GCP

Average in #GCP

quality kandi Quality

pubsub-to-bigquery has 1 bugs (0 blocker, 0 critical, 1 major, 0 minor) and 17 code smells.

pubsub-to-bigquery Quality

Best in #GCP

Average in #GCP

pubsub-to-bigquery Quality

Best in #GCP

Average in #GCP

security Security

pubsub-to-bigquery has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
pubsub-to-bigquery code analysis shows 1 unresolved vulnerabilities (1 blocker, 0 critical, 0 major, 0 minor).
There are 0 security hotspots that need review.

pubsub-to-bigquery Security

Best in #GCP

Average in #GCP

pubsub-to-bigquery Security

Best in #GCP

Average in #GCP

license License

pubsub-to-bigquery is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

pubsub-to-bigquery License

Best in #GCP

Average in #GCP

pubsub-to-bigquery License

Best in #GCP

Average in #GCP

build Reuse

pubsub-to-bigquery releases are not available. You will need to build from source code and install.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
It has 415 lines of code, 5 functions and 3 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

pubsub-to-bigquery Reuse

Best in #GCP

Average in #GCP

pubsub-to-bigquery Reuse

Best in #GCP

Average in #GCP

Top functions reviewed by kandi - BETA

kandi has reviewed pubsub-to-bigquery and discovered the below as its top functions. This is intended to give you an instant insight into pubsub-to-bigquery implemented functionality, and help decide if they suit your requirements.

Parses the XML document from the provided parameters .

Entry point for testing .

Get all kandi verified functions for this library.

pubsub-to-bigquery Key Features

A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub

pubsub-to-bigquery Examples and Code Snippets

See all related Code Snippets

PubSubToBigQuery

Copy

Download

java.exe -jar "C:\Jars\pubsub-to-bq.jar" --runner=BlockingDataflowPipelineRunner --params="&lt;params&gt;&lt;workingBucket&gt;gs://your_bucket&lt;/workingBucket&gt;&lt;maxNumWorkers&gt;1&lt;/maxNumWorkers&gt;&lt;diskSizeGb&gt;250&lt;/diskSizeGb&gt;&lt;machineType&gt;n1-standard-1&lt;/machineType&gt;&lt;keyFile&gt;C:\KeyFiles\YourFile.json&lt;/keyFile&gt;&lt;accountEmail&gt;your_account@developer.gserviceaccount.com&lt;/accountEmail&gt;&lt;projectId&gt;your_project_id&lt;/projectId&gt;&lt;pipelineName&gt;your_pipeline_name&lt;/pipelineName&gt;&lt;pubSubTopic&gt;your_pub_topic&lt;/pubSubTopic&gt;&lt;bqDataSet&gt;your_destination_BQ_dataset&lt;/bqDataSet&gt;&lt;bqTable&gt;your_destination_BQ_table&lt;/bqTable&gt;&lt;streaming&gt;true&lt;/streaming&gt;&lt;zone&gt;us-west1-a&lt;/zone&gt;&lt;schema&gt;{"fields":[{"description":null,"fields":null,"mode":"REQUIRED","name":"Student_Name","type":"STRING","ETag":null}],"ETag":null}&lt;/schema&gt;&lt;/params&gt;"

GCloud Dataflow recreate BigQuery table if it gets deleted during job run

Copy

Download

    public TableReference getOrCreateTable(BigQueryOptions options, String tableSpec)
        throws IOException {
      TableReference tableReference = parseTableSpec(tableSpec);
      if (!createdTables.contains(tableSpec)) {
        synchronized (createdTables) {
          // Another thread may have succeeded in creating the table in the meanwhile, so
          // check again. This check isn't needed for correctness, but we add it to prevent
          // every thread from attempting a create and overwhelming our BigQuery quota.
          if (!createdTables.contains(tableSpec)) {
            TableSchema tableSchema = JSON_FACTORY.fromString(jsonTableSchema, TableSchema.class);
            Bigquery client = Transport.newBigQueryClient(options).build();
            BigQueryTableInserter inserter = new BigQueryTableInserter(client);
            inserter.getOrCreateTable(tableReference, WriteDisposition.WRITE_APPEND,
                CreateDisposition.CREATE_IF_NEEDED, tableSchema);
            createdTables.add(tableSpec);
          }
        }
      }
      return tableReference;
    }

if (!createdTables.contains(tableSpec)) {

    public TableReference getOrCreateTable(BigQueryOptions options, String tableSpec)
        throws IOException {
      TableReference tableReference = parseTableSpec(tableSpec);
      if (!createdTables.contains(tableSpec)) {
        synchronized (createdTables) {
          // Another thread may have succeeded in creating the table in the meanwhile, so
          // check again. This check isn't needed for correctness, but we add it to prevent
          // every thread from attempting a create and overwhelming our BigQuery quota.
          if (!createdTables.contains(tableSpec)) {
            TableSchema tableSchema = JSON_FACTORY.fromString(jsonTableSchema, TableSchema.class);
            Bigquery client = Transport.newBigQueryClient(options).build();
            BigQueryTableInserter inserter = new BigQueryTableInserter(client);
            inserter.getOrCreateTable(tableReference, WriteDisposition.WRITE_APPEND,
                CreateDisposition.CREATE_IF_NEEDED, tableSchema);
            createdTables.add(tableSpec);
          }
        }
      }
      return tableReference;
    }

if (!createdTables.contains(tableSpec)) {

See all related Code Snippets

Community Discussions

Trending Discussions on pubsub-to-bigquery

Unable to drain/cancel Dataflow job. It keeps pending state
GCloud Dataflow recreate BigQuery table if it gets deleted during job run

Trending Discussions on pubsub-to-bigquery

QUESTION

Unable to drain/cancel Dataflow job. It keeps pending state

Asked 2021-Feb-10 at 20:19

Some jobs are remaining with pending pending state and I can't cancel them.

How do I cancel the job.

Web console shows like this.

"The graph is still being analyzed."
All logs are "No entries found matching current filter."
Job status: "Starting..." There isn't appered a cancel button yet.

There are no instances in the Compute Engline tab.

What I did is below. I created a streaming job. it was simple template job, Pubsub subscription to BigQuery. I set machineType as e2-micro because it was just a testing.

I also tried to drain and cancel by gcloud but it doesn't work.

$ gcloud dataflow jobs drain --region asia-northeast1 JOBID

Failed to drain job [...]: (...): Workflow modification failed. Causes: (...): 
Operation drain not allowed for JOBID. 
Job is not yet ready for draining. Please retry in a few minutes. 
Please ensure you have permission to access the job and the `--region` flag, asia-northeast1, matches the job's
region.

This is jobs list

$ gcloud dataflow jobs list --region asia-northeast1
JOB_ID  NAME                               TYPE       CREATION_TIME        STATE      REGION
JOBID1  pubsub-to-bigquery-udf4            Streaming  2021-02-09 04:24:23  Pending    asia-northeast1
JOBID2  pubsub-to-bigquery-udf2            Streaming  2021-02-09 03:20:35  Pending    asia-northeast1
...other jobs...

Please let me know how to stop/cancel/delete these streaming jobs.

Job IDs:

2021-02-08_20_24_22-11667100055733179687
2021-02-08_20_24_22-11667100055733179687

WebUI: https://i.stack.imgur.com/B75OX.png

https://i.stack.imgur.com/LzUGQ.png

ANSWER

Answered 2021-Feb-10 at 12:47

In GCP console Dataflow UI, if you have running Dataflow jobs, you will see the "STOP" button just like the below image.

Press the STOP button.

When you successfully stop your job, you will see the status like below. (I was too slow to stop the job with the first try, so I had to test it again. :) )

Source https://stackoverflow.com/questions/66116354

QUESTION

GCloud Dataflow recreate BigQuery table if it gets deleted during job run

Asked 2020-Mar-22 at 14:36

I have set up a GCloud Dataflow pipeline which consumes messages from a Pub/Sub subscription, converts them to table rows and writes those rows to a corresponding BigQuery table.

Table destination is decided based on the contents of the Pub/Sub message and will occasionally lead to the situation that a table does not exist yet and has to be created first. For this I use create disposition CREATE_IF_NEEDED, which works great.

However, I have noticed that if I manually delete a newly created table in BigQuery while the Dataflow job is still running, Dataflow will get stuck and will not recreate the table. Instead I get an error:

Operation ongoing in step write-rows-to-bigquery/StreamingInserts/StreamingWriteTables/StreamingWrite for at least 05m00s without outputting or completing in state finish at sun.misc.Unsafe.park(Native Method) at
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
    java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429) at
    java.util.concurrent.FutureTask.get(FutureTask.java:191) at
    org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:816) at
    org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:881) at
    org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:143) at
    org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:115) at
    org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)

If I go back to BigQuery and manually recreate this table, Dataflow job will continue working.

However, I am wondering if there is a way to instruct the Dataflow pipeline to recreate the table if it gets deleted during the job run?

GCloud Dataflow pipeline

ANSWER

Answered 2020-Mar-22 at 14:36

This is not possible in the current BigqueryIO connector. From the github link of the connector present here you will observe that for StreamingWriteFn which is what your code, the table creation process is done in getOrCreateTable and this is called in finishBundle. There is a map of createdTables that is maintained and in finishBundle the table gets created if it not is already present, once it is present and stored in the hashmap it is not re-created as shown below:-

    public TableReference getOrCreateTable(BigQueryOptions options, String tableSpec)
        throws IOException {
      TableReference tableReference = parseTableSpec(tableSpec);
      if (!createdTables.contains(tableSpec)) {
        synchronized (createdTables) {
          // Another thread may have succeeded in creating the table in the meanwhile, so
          // check again. This check isn't needed for correctness, but we add it to prevent
          // every thread from attempting a create and overwhelming our BigQuery quota.
          if (!createdTables.contains(tableSpec)) {
            TableSchema tableSchema = JSON_FACTORY.fromString(jsonTableSchema, TableSchema.class);
            Bigquery client = Transport.newBigQueryClient(options).build();
            BigQueryTableInserter inserter = new BigQueryTableInserter(client);
            inserter.getOrCreateTable(tableReference, WriteDisposition.WRITE_APPEND,
                CreateDisposition.CREATE_IF_NEEDED, tableSchema);
            createdTables.add(tableSpec);
          }
        }
      }
      return tableReference;
    }

For you to meet your requirement you might have to maintain your own BigqueryIO wherein you don't perform this specific check

if (!createdTables.contains(tableSpec)) {

The more important question though is why would the table get deleted in a production system by itself? This problem should be fixed rather than trying to re-create the table from Dataflow.

Source https://stackoverflow.com/questions/60786744

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install pubsub-to-bigquery

You can download it from GitHub.
You can use pubsub-to-bigquery like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the pubsub-to-bigquery component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .