exatorrent | Use Torrent Client | Stream Processing library

Any Kafka library should work with RedPanda, including Kafka Streams, KSQL, Apache Spark, Flink, Storm, etc.

You could use TumblingEventTimeWindows of(Time size, Time offset, WindowStagger windowStagger) with WindowStagger.RANDOM.

See https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/assigners/WindowStagger.html for documentation.

Turns out the watermark was being generated after the source was exhausted(in this case it was from reading a file). So the max watermark was emitted(9223372036854775807). A trigger happens when: window.maxTimestamp() <= ctx.getCurrentWatermark()

See https://stackoverflow.com/a/51554273/1099123

That's because integerStream.filter(x -> x > 5); returns a new stream that you ignore.

This works:

There is also a syntax error in your first example. It has an extra semicolon at the end of the fourth line.

In flink, this can be done with state.

Your use case is very similar to the fraud detection example from flink doc.

The pipeline you've described doesn't use any keyed state that would benefit from setting state TTL. The only keyed state in your pipeline is the contents of the session windows, and that state is being purged as soon as possible -- as the sessions close. (Furthermore, since you are using a reduce function, that state consists of just one value per key.)

For the most part, expiring state is only relevant for state you explicitly create, in which case you will have ready access to the state descriptor and can configure it to use State TTL. Flink SQL does create state on your behalf that might not automatically expire, in which case you will need to use Idle State Retention Time to configure it. The CEP library also creates state on your behalf, and in this case you should ensure that your patterns either eventually match or timeout.

After looking at the right place, I found the related configuration: topology.builtin.metrics.bucket.size.secs: 10 is they way to specify that interval in storm.yaml.

You could disable the chaining via (disableChaining()) and start a new chain to isolate it from others via (startNewChain()). You can play with Flink Plan Visualizer to see if your plan has isolated operators. These modifiers applied affter the operator. Example:

Why do you need to isolate it? Well... at the end of any chain flink does a checkpoint (if enabled) and checkpoint should be confirmed (persisted/serialized). Otherwise the system will rollback it and start the process again. For this Flink needs to be sure that it has enough slots beforehand. In your case enough exceptional slots. And if not, the whole stream will be inactive. Therefore you can NOT tell flink that for operator x you need to use only slot X and for operator Z only Y as for Flink is just a computer power which produces intermediate results for the checkpoint (or directly to the next operator).

Stream Processing is based on the fundamental concept of unbounded streams of events (in contrast to static sets of bounded data as we typically find in relational databases).

Taking that unbounded stream of events, we often want to do something with it. An unbounded stream of events could be temperature readings from a sensor, network data from a router, order from an e-commerce system, and so on.

Let's imagine we want to take this unbounded stream of events, perhaps its manufacturing events from a factory about 'widgets' being manufactured.

We want to filter that stream based on a characteristic of the 'widget', and if it's red route it to another stream. Maybe that stream we'll use for reporting, or driving another application that needs to respond to only red widgets events:

This, in a rather crude nutshell, is stream processing. Stream processing is used to do things like:

• filter streams
• aggregate (for example, the sum of a field over a period of time, or a count of events in a given window)
• enrichment (deriving values within a stream of a events, or joining out to another stream)

As you mentioned, there are a large number of articles about this; without wanting to give you yet another link to follow, I would recommend this one.

Kafka Streams a stream processing library, provided as part of Apache Kafka. You use it in your Java applications to do stream processing.

In the context of the above example it looks like this:

Kafka Streams is built on top of the Kafka producer/consumer API, and abstracts away some of the low-level complexities. You can learn more about it in the documentation.

One way of doing this in Flink might be to use a KeyedProcessFunction, i.e. a function that can:

• process each event in your stream
• maintain some state
• trigger some logic with a timer based on event time

So it would go something like this:

• you need to know some kind of "max out of orderness" about your data. Based on your description, let's assume 100ms for example, such that when processing data at timestamp 1612103771212 you decide to consider you're sure to have received all data until 1612103771112.
• your first step is to keyBy() your stream, keying by particle id. This means that the logic of next operators in your Flink application can now be expressed in terms of a sequence of events of just one particle, and each particle is processed in this manner in parallel.

Something like this:

• during the initialization of the ProcessFunction of YourProcessFunction (i.e. during the open method), initialize a ListState where you can safely store stuff.
• when processing an element in your stream, in the processElement method, just add it to the listState and register a timer trigger in, say, 100ms
• when the onTimer() method triggers, say at time t, look at all elements in the listState that have a time < t - 100 and, if you have at least two of them, sort them, remove them from the state, apply the speed calculation and annotation logic you described, and emit the result downstream.

You'll find an example in the official Flink training that is using this kind of logic for the duration of taxi rides, which has lot's of similarities with your use case. Have also a look at the various Readme.md files of that repo for more details.