reactor-kafka | Reactive Kafka Driver with Reactor | Pub Sub library

[Question posted by a user on YugabyteDB Community Slack]

I am currently using YugabyteDB with the reactive Postgres driver (io.r2dbc:r2dbc-postgresql), and I am facing some intermittent issues like this one, as you can see in the stack trace below. I was told that the Postgres driver may not deal correctly with the Yugabyte load balancing, which maybe is leading me to this problem, and then maybe the actual Yugabyte driver would deal properly with such scenarios. However, I am using a reactive code, which means I need an R2DBC driver, and I did not find any official R2DBC Yugabyte driver.

Do you think a more appropriate driver would really solve such problem? If so, is there any other R2DBC driver that would be more suitable for my purpose here? If not, do you have any suggestions to solve the problem below?

I have a reactive kafka application that reads data from a topic and writes to another topic. The topic has multiple partitions and I want to create the same number of consumers(in the same consumer group) as the partitions in the topic. From what I understand from this thread .receive() will create only one instance of KafkaReceiver that will read from all the partitions in the topic. So I would need multiple receivers to read from different partitions in parallel.

To do that I came up with the following code:

I am trying on reactor-kafka for consuming messages. Everything else work fine, but I want to add a retry(2) for failing messages. spring-kafka already retries failed record 3 times by default, I want to achieve the same using reactor-kafka.

I am using spring-kafka as a wrapper for reactive-kafka. Below is my consumer template:

I am facing an issue while unsubscribing from the Kafka consumer, please note that we are using reactor kafka API. It does unsubscribe successfully in the beginning but then it joins the group immediately and is assigned that topic's partitions so essentially it remains subscribed all the time and keeps consuming messages from topic even when it is not supposed to!

Following is my code for doing this business,

I need to consume from a Kafka topic that will have millions of data. Once I read from the topic, i need to transform and write it to another topic. I am able to consume messages from the topic, process the data by multiple threads and write to another topic. I followed the example from here https://projectreactor.io/docs/kafka/1.3.5-SNAPSHOT/reference/index.html#concurrent-ordered

Here is my code:

I am using spring-kafka to implement a consumer that reads messages from a certain topic. All of these messages are processed by them being exported into another system via a REST API. For that, the code uses the WebClient from the Spring Webflux project, which results in reactive code:

Below is a sample code which uses reactor-kafka and reads data from a topic (with retry logic) which has records published via a non-reactive producer. Inside my doOnNext() consumer I am using non-reactive elasticsearch client which indexes the record in the index. So I have few questions that I am still unclear about :

1. I know that consumers and producers are independent decoupled systems, but is it recommended to have reactive producer as well whose consumers are reactive?
2. If I am using something that is non-reactive, in this case Elasticsearch client org.elasticsearch.client.RestClient, does the "reactiveness" of the code work? If it does or does not, how do I test it? (By "reactiveness", I mean non blocking IO part of it i.e. if I spawn three reactive-consumers and one is latent for some reason, the thread should be unblocked and used for other reactive consumer).
3. In general the question is, if I wrap some API with reactive clients should the API be reactive as well?

i am asking myself if the ReactiveKafkaConsumerTemplate of the spring-kafka project does guarantee the correct ordering of messages. I read the documentation of the reactor-kafka project and it states that messages should be consumed using the concatMap operator, but the ReactiveKafkaConsumerTemplate uses the flatMap operator at least in case of the receiveAutoAck method here:

https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/core/reactive/ReactiveKafkaConsumerTemplate.java#L69

Reference documentation of the reactor-kafka project: https://projectreactor.io/docs/kafka/release/reference/#_auto_acknowledgement_of_batches_of_records

I am interested in using receiveAutoAck as it seems to be the most simpelst and comfortable approach, which suffices my use case. The only way to overcome this behaviour of the receiveAutoAck method seems to subclass the ReactiveKafkaConsumerTemplate and overwrite this behaviour. Is this correct?

I have task that runs a functional test