kafka-tutorial | Kafka Tutorial for https : //dn.dev/kafkamaster | Stream Processing library

You cannot specify partitions in the Connect API. It subscribes to all partitions, then distributes consumer instances amongst worker tasks as part of a consumer group.

You are correct - messages need to be deserialized, then inspected against a predicate (in application space)

throw away those that are not relevant, I could do that by hand

Sure, you could, but Kafka Streams has useful methods for defining session windows. Plus, you wouldn't need to define a consumer and producer instance to forward to new topics.

To inform all fellows. If you use a insert into select...-statement, a running query will be created and the final stream will receive further updates.

Based on this post https://www.nerd.vision/post/suppress-surprise-kafka-streams-and-the-suppress-operator

The suppress operator is based on event-time and as long as no new records arrive the stream is basically frozen.

This post explains how to test this.

For the tests to work you need:

1. produce test data
2. produce a dummy event with future timestamp to release the window result assert.

Note that each test needs to be isolated (e.g bring Kafka broker and the stream up before and turn off after each individual test or close the test driver).

If you already have a topic, you should skip to step 3 where it actually sets up the topics to query via Presto

kafka-tpch load creates new topics with the specified prefix

I’m sorry for the trouble. This is indeed a tricky problem. I have some ideas for adding some operations to support this kind of integration testing, but it’s hard to do without breaking basic stream processing time semantics.

It sounds like you’re testing a “real” KafkaStreams application, as opposed to testing with TopologyTestDriver. My first suggestion is that you’ll have a much better time validating your application semantics with TopologyTestDriver, if it meets your needs.

It sounds to me like you might have more than one partition in your input topic (and therefore your application). In the event that key 1 goes to one partition, and key 3 goes to another, you would see what you’ve observed. Each partition of your application tracks stream time independently. TopologyTestDriver works nicely because it only uses one partition, and also because it processes data synchronously. Otherwise, you’ll have to craft your “dummy” time advancement messages to go to the same partition as the key you’re trying to flush out.

This is going to be especially tricky because your “flatMap().groupByKey()” is going to repartition the data. You’ll have to craft the dummy message so that it goes into the right partition after the repartition. Or you could experiment with writing your dummy messages directly into the repartition topic.

If you do need to test with KafkaStreams instead of TopologyTestDriver, I guess the easiest thing is just to write a “time advancement” message per key, as you were suggesting in your question. Not because it’s strictly necessary, but because it’s the easiest way to meet all these caveats. I’ll also mention that we are working on some general improvements to stream time handling in Kafka Streams that should simplify the situation significantly, but that doesn’t help you right now, of course.

Ok, it looks like setting --timeout=0 in the producer makes the lag disappear. Looking at the kafka-console-producer source code, --timeout defaults to 1000 and gets merged into LINGER_MS_CONFIG. So even though linger defaults to zero in Kafka generally, it effectively defaults to 1 sec in this command line producer.