scala-kafka | Quick up and running using Scala for Apache Kafka | Pub Sub library

As far as Kafka is concerned, the message is consumed as soon as Alpakka Kafka reads it from Kafka.

This is before the actor inside of Alpakka Kafka has emitted it to a downstream consumer for application level processing.

Kafka auto-commit (enable.auto.commit = true) will thus result in the offset being committed before the message has been sent to your actor.

The Kafka docs on offset management do (as of this writing) refer to enable.auto.commit as having an at-least-once semantic, but as noted in my first paragraph, this is an at-least-once delivery semantic, not an at-least-once processing semantic. The latter is an application level concern, and accomplishing that requires delaying the offset commit until processing has completed.

The Alpakka Kafka docs have an involved discussion about at-least-once processing: in this case, at-least-once processing will likely entail introducing manual offset committing and replacing mapAsyncUnordered with mapAsync (since mapAsyncUnordered in conjunction with manual offset committing means that your application can only guarantee that a message from Kafka gets processed at-least-zero times).

In Alpakka Kafka, a broad taxonomy of message processing guarantees:

• hard at-most-once: Consumer.atMostOnceSource - commit after every message before processing
• soft at-most-once: enable.auto.commit = true - "soft" because the commits are actually batched for increased throughput, so this is really "at-most-once, except when it's at-least-once"
• hard at-least-once: manual commit only after all processing has been verified to succeed
• soft at-least-once: manual commit after some processing has been completed (i.e. "at-least-once, except when it's at-most-once")
• exactly-once: not possible in general, but if your processing has the means to dedupe and thus make duplicates idempotent, you can have effectively-once