kafka-proxy | Proxy connections to Kafka cluster | Pub Sub library

An answer was provided by Alan Klikic on Lightbend discussion forums here.

Part 1:

If you are only using external Kafka cluster in your business service then you can implement this using only Lagom Broker API. So you need to:

1. create API with service descriptor with only topic definition (this API is not beeing implemented)
2. in your business service configure kafka_native depending on your deployment (as i mentioned in previous post)
3. in your business service inject service from API created in #1 and subscribe to it using Lagom Broker API subscriber

Offset commiting, in Lagom Broker API subscriber is handled out-of-the-box.

Part 2:

Kafka and AMQP consumer implementations require persistant akka stream. So you need to handle disconnects. These can be done in two ways:

1. control peristant akka stream by wraping it in an actor. You initialize you stream Flow on actor preStart and pipe stream complete to the actor that will stop it. If stream completes or fails actor will stop. Then wrap actor in actor backoff with restart strategy, that will restart the actor in case of complete or fail and reinitialize the Flow
2. akka streams Delayed restarts with backoff stage

Personnaly I use #1 and did not try #2 yet.

Initializing backoff actor for #1 or Flow for #2 can be done in your Lagom components trait (basically in the same place where you do your subscribe now using Lagom Broker API).

Be sure to set a consumer group when configuring consumer to ensure avoiding duplicate consuming. You can use, like Lagom does, service name from descriptor as consumer group name.