pcollections | A Persistent Java Collections Library | Hashing library

When using the Apache Beam model, it is a common misconception that the parallelization scheme is defined by the PCollection (understandable since this is short for Parallel Collection). In actuality, the parallelization is defined per key in each PCollection[1]. In other words, the Beam model processes keys in parallel but values in a single key sequentially.

The problem you are coming up against is commonly referred to as a hot key. This happens when too many values are paired to a single key, thus limiting parallelism.

To manipulate the data to the expected output you will have to edit your existing pipeline to emit the values in such a way that not all elements go to a single key. This is a little tough because it looks like in your example you wish to output the index with the element. If this is the case, then no matter how you cut it, you will have to merge all the values somewhere to a key in memory to get the correct index.

If you don't care about getting the specific index like you have in the above example then take a look at the following code. This code assigns each element to a random partition within each key. This helps to break up the number of elements per key into something manageable.

You can read the beam programming guide. You will find great solution and pattern for your use case. For example, to filter the good and the bad JSON, you need to create a transform with a standard output (let's say the correct JSON) and an addition output for the bad JSON.

So, from there, you have 2 PCollections. Then process them idependently. You can sink the bad JSON in a file, in BigQuery, or simply create a transform that write a special log trace in Cloud Logging to get and process this log trace later in another process if you want.

The statement on PCollection immutability is that DoFns SHOULD not mutate their inputs. In languages like Python where everything is passed by reference and by default mutable (no const), this is difficult if not impossible to enforce.

The parquet format is optimized for writing data in batches. Therefore it doesn't lend itself well to streaming, where you receive records one by one. In your example you're writing row by row in a parquet file, which is super unefficient.

I'd recommand saving your data in a format that lends itself well to appending data row by row, and then have a regular job that moves this data in batches to parquet files.

Or you can do like apache_beam.io.parquetio._ParquetSink. It keeps records in memory in a buffer and write them in batch every now and then. But with this you run the risk of losing the records in the buffer if your application crashes.

This question is linked to this previous question which contains more detail about the implementation. The solution there suggested to create an instance of google.cloud.storage.Client() in the start_bundle() of every call to a ParDo(DoFn). This is connected to the same gcs bucket - given via the args in WriteToText(known_args.output)

The | represents a separation between steps, this is (using p as Pbegin): p | ReadFromText(..) | ParDo(..) | GroupByKey().

You can also reference other PCollections before |:

ParDo specifies a generic parallel processing and the runner will manage this "fanout", while Partition has no intention for a parallel but it is designed for spliting a collection into a sequence of sub-collections with the logic determined by a function which you create.

A typical user case for partition can be that of grouping students by percentile and passing groups to their corresponding downstream steps. Notice that different group of student can have different downstream process, and this is just not what ParDo is designed for.

In addition, another difference between Partition and ParDo is that the former must have a predefined partition number while the latter has no such concept.

You are using a non-deterministic triggering, which means the output is sensitive to the exact ordering in which events come in. Another way to look at this is that CoGBK does not wait for both sides to come in; the trigger starts ticking as soon as either side comes in.

For example, lets call your PCollections A and A' respectively, and assume they each have two elements a1, a2, a1', and a2' (of common provenance).

Suppose a1 and a1' come into the CoGBK, 39 seconds passes, and then a2 comes in (on the same key), another 2 seconds pass, then a2' comes in. The CoGBK will output ([a1, a2], [a1']) when the 40-second mark hits, and then when the window closes ([], [a2']) will get emitted. (Even if everything is on the same key, this could happen occasionally if there is more than a 40-second walltime delay going through the longer path, and will almost certainly happen for any late data (each side will fire separately).

Draining makes things worse, e.g. I think all processing time triggers fire immediately.

Based on the resulting table you showed, I assume you want your output to look like this: