scio | A Scala API for Apache Beam and Google Cloud Dataflow | GCP library

I'm using Apache Beam 2.28.0 on Google Cloud DataFlow (with Scio SDK). I have a large input PCollection (bounded) and I want to limit / sample it to a fixed number of elements, but I want to start the downstream processing as soon as possible.

Currently, when my input PCollection has e.g. 20M elements and I want to limit it to 1M by using https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/transforms/Sample.html#any-long-

I'm trying to use scio_beast in a project. I understand its rather unfinished, but that should not matter much. I've managed to get it working pretty well.

I'm trying to connect to a server behind CloudFlare now, an I understand I need SNI for that to work.

Given the following:

I'm having a question while I'm writing code with Apache-beam using Dataflow.

Originally, I wrote code with python, but I checked java, go, and scio among the supported languages.

Please give us feedback on whether there is a language that has the best performance.

Or is there more library support?

It's my personal curiosity, but it's hard to summarize the contents in the document, so I wrote a question. Thank you.

For example, the background-Mnist that I loaded from a mat file gives 50,000x784 for the training set.

There should be 50,000 of 28x28 image

I reshaped the whole thing using

I'm trying to aggregate (per key) a streaming data source in Apache Beam (via Scio) using a stateful DoFn (using @ProcessElement with @StateId ValueState elements). I thought this would be most appropriate for the problem I'm trying to solve. The requirements are:

• for a given key, records are aggregated (essentially summed) across all time - I don't care about previously computed aggregates, just the most recent
• keys may be evicted from the state (state.clear()) based on certain conditions that I control
• Every 5 minutes, regardless if any new keys were seen, all keys that haven't been evicted from the state should be outputted

Given that this is a streaming pipeline and will be running indefinitely, using a combinePerKey over a global window with accumulating fired panes seems like it will continue to increase its memory footprint and the amount of data it needs to run over time, so I'd like to avoid it. Additionally, when testing this out, (maybe as expected) it simply appends the newly computed aggregates to the output along with the historical input, rather than using the latest value for each key.

My thought was that using a StatefulDoFn would simply allow me to output all of the global state up until now(), but it seems this isn't a trivial solution. I've seen hintings at using timers to artificially execute callbacks for this, as well as potentially using a slowly growing side input map (How to solve Duplicate values exception when I create PCollectionView>) and somehow flushing this, but this would essentially require iterating over all values in the map rather than joining on it.

I feel like I might be overlooking something simple to get this working. I'm relatively new to many concepts of windowing and timers in Beam, looking for any advice on how to solve this. Thanks!

I got this in my .proto file:

Is there any way to view the contents of an SCollection when running a unit test (PipelineSpec)?

When running something in production on many machines there would be no way to see the entire collection in one machine, but I wonder is there a way to view the contents of an SCollection (for example when running a unit test in debug mode in intellij).