stream-processing | Stream processing guidelines and examples | Stream Processing library

Stream Processing is based on the fundamental concept of unbounded streams of events (in contrast to static sets of bounded data as we typically find in relational databases).

Taking that unbounded stream of events, we often want to do something with it. An unbounded stream of events could be temperature readings from a sensor, network data from a router, order from an e-commerce system, and so on.

Let's imagine we want to take this unbounded stream of events, perhaps its manufacturing events from a factory about 'widgets' being manufactured.

We want to filter that stream based on a characteristic of the 'widget', and if it's red route it to another stream. Maybe that stream we'll use for reporting, or driving another application that needs to respond to only red widgets events:

This, in a rather crude nutshell, is stream processing. Stream processing is used to do things like:

• filter streams
• aggregate (for example, the sum of a field over a period of time, or a count of events in a given window)
• enrichment (deriving values within a stream of a events, or joining out to another stream)

As you mentioned, there are a large number of articles about this; without wanting to give you yet another link to follow, I would recommend this one.

Kafka Streams a stream processing library, provided as part of Apache Kafka. You use it in your Java applications to do stream processing.

In the context of the above example it looks like this:

Kafka Streams is built on top of the Kafka producer/consumer API, and abstracts away some of the low-level complexities. You can learn more about it in the documentation.

The amount of channels has no impact on the performance. However, i wonder what is it that you are creating that requires that many inputs and outputs. That looks definitely like an anti-pattern for microservices. In other words, i see an architectural issue here, so feel free to share your business requirement and perhaps we can help you find a better solution.

Also, you are using a very outdated API. The annotation such as @Input/@Output, @EnableBinding, @StreamListener etc being deprecated. We have switched to a simpler functional model.

You can also read these two blog posts for additional context:

• https://spring.io/blog/2019/10/14/spring-cloud-stream-demystified-and-simplified
• https://spring.io/blog/2019/10/17/spring-cloud-stream-functional-and-reactive

I just went through pages 1 at a time and printed(you can use f.write). You need to add a time.sleep() so Selenium doesn't crash. This can go forever until it runs out of pages. Or if specify a if page==n: break. It can even go through the seleniums life cycle.

Rekeying seems to be the right approach, however, you cannot convert a STREAM into a TABLE directly.

Note, that your rekeyed stream customers_stream2 is written into a corresponding topic. Hence, you should be able to crate a new TABLE from the stream's topic to get the latest value per key.

You mix de/serialization. Since you configure the consumer, you need to use only proper deserialization interfaces and implementations:

It's a good question in that we hear of in-memory Spark vs. Hadoop, so a little confusing. The docs are terrible, but I ran a few things and verified observations by looking around to find a most excellent source: http://hydronitrogen.com/apache-spark-shuffles-explained-in-depth.html

Assuming an Action has been called - so as to avoid the obvious comment if this is not stated, assuming we are not talking about ResultStage and a broadcast join, then we are talking about ShuffleMapStage. We look at an RDD initially.

Then, borrowing from the url:

• DAG dependency involving a shuffle means creation of a separate Stage.
• Map operations are followed by Reduce operations and a Map and so forth.

CURRENT STAGE

• All the (fused) Map operations are performed intra-Stage.
• The next Stage requirement, a Reduce operation - e.g. a reduceByKey, means the output is hashed or sorted by key (K) at end of the Map operations of current Stage.
• This grouped data is written to disk on the Worker where the Executor is - or storage tied to that Cloud version. (I would have thought in memory was possible, if data is small, but this is an architectural Spark approach as stated from the docs.)
• The ShuffleManager is notified that hashed, mapped data is available for consumption by the next Stage. ShuffleManager keeps track of all keys/locations once all of the map side work is done.

NEXT STAGE

• The next Stage, being a reduce, then gets the data from those locations by consulting the Shuffle Manager and using Block Manager.
• The Executor may be re-used or be a new on another Worker, or another Executor on same Worker.

So, my understanding is that architecturally, Stages mean writing to disk, even if enough memory. Given finite resources of a Worker it makes sense that writing to disk occurs for this type of operation. The more important point is, of course, the 'Map Reduce' implementation. I summarized the excellent posting, that is your canonical source.

Of course, fault tolerance is aided by this persistence, less re-computation work.

Similar aspects apply to DFs.

cfn-lint warns:

The closest Flink's watermarking comes to supporting this directly is probably the support for per-kafka-partition watermarking -- which isn't really a practical solution to the situation you describe (since having a kafka partition per user isn't realistic).

What can be done is to simply ignore watermarking, and implement the logic yourself, using a KeyedProcessFunction.

BTW, there was recently a thread about this on both the flink-user and flink-dev mailing lists under the subject Per Key Grained Watermark Support.

It is not recommended to run your Streams application on the same servers as your brokers (even if this is technically possible). Kafka's Streams API offers an application-based approach -- not a cluster-based approach -- because it's a library and not a framework.

It is not required to scale your Kafka cluster to scale your Streams application. In general, the parallelism of a Streams application is limited by the number of partitions of your app's input topics. It is recommended to over-partition your topic (the overhead for this is rather small) to guard against scaling limitations.

Thus, it is even simpler to "offer anyone to build applications" as everyone owns their application. There is no need to submit apps to a cluster. They can be executed anywhere you like (thus, each team can deploy their Streams application the same way by which they deploy any other application they have). Thus, you have many deployment options from a WAR file, over YARN/Mesos, to containers (like Kubernetes). Whatever works best for you.

Even if frameworks like Flink, Storm, or Samza offer cluster management, you can only use such tools that are integrated with those frameworks (for example, Samza requires YARN -- no other options available). Let's say you have already a Mesos setup, you can reuse it for your Kafka Streams applications -- no need for a dedicated "Kafka Streams cluster" (because there is no such thing).