Alpakka | Akka Streams Connectors - Alpakka

I would probably tend to implement this by spawning an actor for each websocket, prematerializing a Source which will receive messages from the actor (probably using ActorSource.actorRefWithBackpressure), building a Sink (likely using ActorSink.actorRefWithBackpressure) which adapts incoming websocket messages into control-plane messages (initialization (including the ActorRef associated with the prematerialized source) and transformation changes) and sends them to the actor, and then tying them together using the handleMessagesWithSinkSource on WebsocketUpgrade.

The actor you're spawning would, on receipt of the initialization message, start a stream which is feeding messages to it from Kafka; some backpressure can be fed back to Kafka by having the stream feed messages via an ask protocol which waits for an ack; in order to keep that stream alive, the actor would need to ack within a certain period of time regardless of what the downstream did, so there's a decision to be made around having the actor buffer messages or drop them.

As far as Kafka is concerned, the message is consumed as soon as Alpakka Kafka reads it from Kafka.

This is before the actor inside of Alpakka Kafka has emitted it to a downstream consumer for application level processing.

Kafka auto-commit (enable.auto.commit = true) will thus result in the offset being committed before the message has been sent to your actor.

The Kafka docs on offset management do (as of this writing) refer to enable.auto.commit as having an at-least-once semantic, but as noted in my first paragraph, this is an at-least-once delivery semantic, not an at-least-once processing semantic. The latter is an application level concern, and accomplishing that requires delaying the offset commit until processing has completed.

The Alpakka Kafka docs have an involved discussion about at-least-once processing: in this case, at-least-once processing will likely entail introducing manual offset committing and replacing mapAsyncUnordered with mapAsync (since mapAsyncUnordered in conjunction with manual offset committing means that your application can only guarantee that a message from Kafka gets processed at-least-zero times).

In Alpakka Kafka, a broad taxonomy of message processing guarantees:

• hard at-most-once: Consumer.atMostOnceSource - commit after every message before processing
• soft at-most-once: enable.auto.commit = true - "soft" because the commits are actually batched for increased throughput, so this is really "at-most-once, except when it's at-least-once"
• hard at-least-once: manual commit only after all processing has been verified to succeed
• soft at-least-once: manual commit after some processing has been completed (i.e. "at-least-once, except when it's at-most-once")
• exactly-once: not possible in general, but if your processing has the means to dedupe and thus make duplicates idempotent, you can have effectively-once

The output of notify is a PushResult, but the input of writeOutput is ByteString. Once you change that it will compile. In case you need ByteString, get the same from OutputRecord.

BTW, in the sample code that you have provided, a similar error exists in readCSV and process.

Would sth like this work for you?

The throttle stage in Akka Streams can be used to limit the rate at which elements are passed to the next stage in a stream (using the Scala API):

As far as I know, Alpakka's Spring integration is basically designed around exposing Akka Streams via a Spring HTTP controller. So I'm not sure what purpose bringing Spring into this serves, since there's quite an impedance mismatch between the way an Akka application will tend to like to work and the way a Spring application will tend to like to work.

Assuming you're talking about using Alpakka Kafka, the most idiomatic thing to do would be to just start a stream fed by an Alpakka Kafka Source in your main method and it will run until killed or it fails. You may want to use a RestartSource around the consumer and business logic to ensure that in the event of failure the stream restarts (note that one should generally expect messages for which the offset commit hadn't happened to be processed again, as Kafka in typical cases can only guarantee at-least-once processing).

When you restart the Kafka source after a failure, that results in a new consumer being created; eventually the consumer in the failed source is declared dead by Kafka, triggering a rebalance. In that rebalance, there are no external guarantees of which consumer in the group will be assigned which partition. This would explain why your other consumer in the group reads that partition.

The issue here with a poison message derailing consumption is a major reason I've developed a preference to treat keys and values from Kafka as blobs by using the ByteArrayDeserializer and do the deserialization myself in the stream, which gives me the ability to record (e.g. by logging; producing the message to a dead-letter topic for later inspection can also work) that there was a malformed message in the topic and move on by committing the offset. Either in Scala is particularly good for moving the malformed message directly to the committer.

You are most likely missing hadoop-aws lib on your classpath.

Have a look here: https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html

And also this SO gives some more details how to setup credentials for access to S3: How do I configure S3 access for org.apache.parquet.avro.AvroParquetReader?

Once you have AvroParquetReader correctly initialized, then you can create Akka Stream's Source out of it as per the Alpakka Avro Parquet doc (https://doc.akka.io/docs/alpakka/current/avroparquet.html#source-initiation)

The issue in your code is that the events: Future would only complete when the stream (eventSource) completes.

I'm not familiar with SSE but the stream likely never completes in your case as it's always listening for new events.

You can learn more in Akka Stream documentation.

Depending on what you want to do with the events, you could just map on the stream like: