PAssert | Power Assert inspired debug tool in Swift | Assertion library

I am trying to have a PCollection of AutoValue-defined objects that I have created, and I've added the appropriate annotations to infer the Schema via DefaultSchema(AutoValueSchema.class). Like so:

I'm trying to build a unit test for my pipeline.

This pipeline read from pubsub, perform a transformation and write the result to pubsub again.

To simplify even more the unit test until it works, the unit test will only receive a string as an input and test if the output is the some string.

The code looks like the following:

I am attempting to recreate a simple example of the apache beam pipeline described in this blog post, which makes use of state and timers.

I have written this code to try and test what is in the blog post. The code should simply enrich string records by appending ": enrich" to each record. Before I could add the "stale" timer, I got an error.

I am using apache beam version 2.13, and the direct runner.

Here is the Enrich DoFn that is mostly copy pasted from the blog:

I'm in the process of writing a simple test to verify the semantics of early/on-time/late panes. The pipeline combines the number of elements per key. My early and on-time panes are working as expected, though my final pane seems to be empty at all times.

I have a PCollection which I am certain contains:

• "Bob"
• "John"
• "Fred"

However, when I test an assertion that asks if "Bob" is in the PCollection using:

I am trying to trying to write a Beam Streaming pipeline, that simply reads from a PubSub Queue, parses the data and writes to either one of two BigQuery Tables. So code takes advantage of side outputs to write to one of two tables from within the DoFn. I am running into the following error message: java.lang.IllegalArgumentException: unable to serialize DoFnAndMainOutput{doFn=com.pipeline.PubSubToBigQuery$ParsePubSubMessage@50eca7c6, mainOutputTag=Tag}. I will attach the full error message, DoFn and Test class below:

DoFn:

I may be missing something obvious, but for some reason I can't make PAssert & TestPipeline work with CoGroupByKey -- but without it, it works fine.

Here is a reference test file that can reproduce the issue I'm facing. I tested with both beam sdk 2.4 and 2.5.

For comparison, testWorking works as intended, and testBroken has an additional step like this:

I am writing a pipeline that is processing product events (create, update, delete). Each product belongs to a sale that has a certain duration. I want to be able to perform some aggregation on all the products in a given sale. For the purpose of this example, let's assume I just want a list of unique product IDs per sale.

Therefore, my pipeline is using session windows on the sale id with a very long gap duration (so when the sale closes and there are no more product updates being published, the window for that sale closes too). My question is, how do I write a unit test for that?

For the sake of this test, let's assume the following:

• the events are just Strings with the sale ID and the product ID, separated by a space,
• the applyDistinctProductsTransform will basically perform what I've said above. Create KV elements where the key is the sale id; set session windows with a gap duration of 600 seconds; and finally create a concatenated string of all product IDs per sale.

Here is what I have so far:

I create a TestStream and add some elements: 3 products for sale1. Next, I advance the watermark to 700, well beyond the gap duration. Another product is added and finally the watermark is advanced to infinity.

I am attempting to use Apache Beam from Java as a data pipeline of sorts. I have written a simple class that sources from Google Pubsub and sinks to Google Bigquery, but I cannot get it to build for the life of me. I am using Maven to build and have added every Beam package I could find, but I still get "class file not found" errors.

Specifically: