materializer | Material Design Components for WordPress | Content Management System library

I thought this would be of normal difficulty, I was wrong. One thing I wanted to do is to ensure that the flow counting items that pass through the stream does not keep a reference to each item it sees: if many items pass in the aggregation period, you will end up with an unnecessarily big list in memory (even if only for a second) and the performance penalty to add (many) items to it. The following solution, although complex, keeps only a counter.

NOTE: Although I tested the happy scenario, I cannot say this is battle-proven, so use with caution!

Based on Akka's GroupedWeightedWithin and the documentation here:

I think what you may be missing is to understand the concept of materialized value in Akka Streams. Scan through this part of the documentation especially around combining materialized values. I also had a go at trying to explain this concept here (search for Materialized value). If you grok materialized value, then perhaps what I write here will make more sense.

A call to Source.actorRef(..) returns Source where T is data type of elements flowing through the stream (in your case it's Integer) and ActorRef is the materialized value of that Source. You get the materialized value synchronously when you call run on RunnableGraph which is what to(...) call returns.

That ActorRef is how you can "drive" the stream as per Source.actorRef(...) semantics.

Now the question is how you get your hands on the data that passes through the stream. In your case you're reducing all Integers into one, so instead of using Sink.foreach(...) which is good for side effects, you can use Sink.head. You see,Sinks can also produce materialized values and in case of Sink.head it materializes to CompletionStage of the first element in the stream, which in your case is the only element. So let's try that:

If I understand you correctly you want to have a fixed parallelism across dynamically changing number of Sources that come and go as Kafka is rebalancing topic partitions.

Have a look at first example in the Alpakka Kafka documentation here. It can be adjusted to your example like this:

As I see, this toParquetSingleFile creates a Sink with a Future[Done] as materialized value. But, in your function you are returning via mapMaterializedValue one FileDetails instance. I think that the mapMaterializedValue function that you are using accepts a function of

TL;DR yes, it does. From Stream ordering docs in Akka:

In Akka Streams almost all computation operators preserve input order of elements. This means that if inputs {IA1,IA2,...,IAn} “cause” outputs {OA1,OA2,...,OAk} and inputs {IB1,IB2,...,IBm} “cause” outputs {OB1,OB2,...,OBl} and all of IAi happened before all IBi then OAi happens before OBi.

This property is even upheld by async operations such as mapAsync, however an unordered version exists called mapAsyncUnordered which does not preserve this ordering.

However, in the case of Junctions which handle multiple input streams (e.g. Merge) the output order is, in general, not defined for elements arriving on different input ports. That is a merge-like operation may emit Ai before emitting Bi, and it is up to its internal logic to decide the order of emitted elements. Specialized elements such as Zip however do guarantee their outputs order, as each output element depends on all upstream elements having been signalled already – thus the ordering in the case of zipping is defined by this property.

If you find yourself in need of fine grained control over order of emitted elements in fan-in scenarios consider using MergePreferred, MergePrioritized or GraphStage – which gives you full control over how the merge is performed.

You can accomplish this with delayWith, which allows a (potentially stateful) method to define how long to delay this element (without re-ordering elements), for example:

In test, the println that prints the thread ID is executed outside of the future, therefore it's executed synchronously. The code inside the future will get executed on a thread of the ExecutionContext (in this case the actor system's dispatcher). It's worth noting that some parallel execution happened: the thread print for a = 4 happened before the a + 1 print for a = 3.

If you move the thread println into the future, that println will execute asynchronously:

You can extract to a separate module not Test but TestMacro

core

Materializer is easy, supervision strategy requires more changes.

1. Supervision strategy. You need to provide decider as attribute of a stream, e.g.: