beam.io | Rasperry Pi connected to a Tidmarsh sensor node

I have a Python Apache Beam streaming pipeline running in Dataflow. It's reading from PubSub and writing to GCS. Sometimes I get errors like "Error in _start_upload while inserting file ...", which comes from:

I want to publish messages to a Pub/Sub topic with some attributes thanks to Dataflow Job in batch mode.

My dataflow pipeline is write with python 3.8 and apache-beam 2.27.0

It works with the @Ankur solution here : https://stackoverflow.com/a/55824287/9455637

But I think it could be more efficient with a shared Pub/Sub Client : https://stackoverflow.com/a/55833997/9455637

However an error occurred:

return StockUnpickler.find_class(self, module, name) AttributeError: Can't get attribute 'PublishFn' on

Questions:

1. Would the shared publisher implementation improve beam pipeline performance?
2. Is there another way to avoid pickling error on my shared publisher client ?

My Dataflow Pipeline :

I am facing with a problem in dataflow. I used Python bigquery api, and it works fine with autodetect. It run fine, job_config create table and at the same time append values:

I am having some issues when try to execute a DataFlow job orchestrated by Airflow. After triggered the DAG, i receive this error:

module 'apache_beam.io' has no attribute 'ReadFromBigQuery''

I am trying to read a topic from pubsub and do some cleanup/transfermation and write the final result to another pubsub topic. however i am ending up with the following error. pls guide me.

code:

I have a pipeline as follow:

A little bit of a newbie to Dataflow here, but have succesfully created a pipleine that works well.

The pipleine reads in a query from BigQuery, applies a ParDo (NLP fucntion) and then writes the data to a new BigQuery table.

The dataset I am trying to process is roughly 500GB with 46M records.

When I try this with a subset of the same data (about 300k records) it works just fine and is speedy see below:

When I try run this with the full dataset, it starts super fast, but then tapers off and ultimately fails. At this point the job failed and had added about 900k elements which was about 6-7GB and then the element count actually started decreasing.

I am using 250 workers and a n1-highmem-6 machine type

In the worker logs I get a few of these (about 10):

I have a streaming dataflow pipeline that writes to BQ, and I want to window all the failed rows and do some further analysis. The pipeline looks like this, I'm getting all the error messages in the 2nd step but all the messages are getting stuck to the beam.GroupByKey(). Nothing moves downstream after that. Does anyone have any idea how to fix this?

I'm currently creating a Streaming Dataflow job that only carries out computation if and only if there is an increment in the "Ring" column of my data.

My data flow code