spring-cloud-stream-binder-kafka | Spring Cloud Stream binders for Apache Kafka | Pub Sub library

we are using Spring Cloud Stream to listen to rabbitMQ multiple queues, especially the SCF model

• The spring-cloud-stream-reactive module is deprecated in favor of native support via Spring Cloud Function programming model.

by the time there was a single node/host it was working good (application.yml snippet shared below),

however the moment we try to connect multiple nodes it is failing, Can someone guide how to connect the same or have some sample related to Spring Cloud Documentation

Following Code is working as expected

Initial Question: I have a question how I can bind my GlobalStateStore to a processor. My Application has a GlobalStateStore with an own processor ("GlobalConfigProcessor") to keep the Store up to date. Also, I have another Processor ("MyClassProcessor") which is called in my Consumer Function. Now I try to access the store from MyClassProcessor, but I get an exception saying : Invalid topology: StateStore config_statestore is not added yet.

Update on current situation: I setup a test repository to give a better overview over my situation. This can be found here: https://github.com/fx42/store-example

As you can see in the repo, I have two Consumers which both consume different topics. The Config-Topic provides an event which I want to write to a GlobalStateStore. Here are the StateStoreUpdateConsumer.java and the StateStoreProcessor.java involved. With the MyClassEventConsumer.java I process another Input-Topic and want to read values from the GlobalStateStore. As provided in this doc I can't initialize GlobalStateStores just as StateStoreBean but instead I have to add this actively with the StreamsBuilderFactoryBeanCustomizer Bean. This Code is currently commented out in the StreamConfig.java. Without this code I get the Exception

I'm trying to setup a spring cloud stream project with kafka. Everything is working as expected but the key de/serialization. The files are the pom.xml:

I'm trying to use Spring Cloud Stream to process messages sent to an Azure Event Hub instance. Those messages should be routed to a tenant-specific topic determined at runtime, based on message content, on a Kafka cluster. For development purposes, I'm running Kafka locally via Docker. I've done some research about bindings not known at configuration time and have found that dynamic destination resolution might be exactly what I need for this scenario.

However, the only way to get my solution working is to use StreamBridge. I would rather use the dynamic destination header spring.cloud.stream.sendto.destination, in that way the processor could be written as a Function<> instead of a Consumer<> (it is not properly a sink). The main concern about this approach is that, since the final solution will be deployed with Spring Data Flow, I'm afraid I will have troubles configuring the streams if using StreamBridge.

Moving on to the code, this is the processor function, I stripped away the unrelated parts

In the event you only have single bean of type java.util.function.[Supplier/Function/Consumer], you can skip the spring.cloud.function.definition property, since such functional bean will be > > auto-discovered. However, it is considered best practice to use such property to avoid any confusion.

So I do have multiple beans in my project but still only one bean of type supplier.

Not sure what exactly what I am missing.

EXCEPTION TRACE:

My understanding was that spring-kafka was created to interact with Kafka Client APIs, and later on, spring-cloud-stream project was created for "building highly scalable event-driven microservices connected with shared messaging systems", and this project includes a couple of binders, one of them is a binder that allows the interaction with Kafka Stream API:

I am trying to write a spring-cloud-stream function (spring-starter-parent 2.5.3, java 11, spring-cloud-version 2020.0.3) which has both a Kafka and Postgres transaction. The function will raise a simulated error whenever the consumed message starts with the string "fail," which I expect to cause the database transaction to roll back, then cause the kafka transaction to roll back. (I am aware that the Kafka transaction is not XA, which is fine.) So far I have not gotten the database transaction to work, but the kafka transaction does.

Currently I am using a @Transactional annotation, which does not appear to start a database transaction. (The Kafka binder documentation recommends synchronizing database + kafka transactions using the ChainedTransactionManager, but the Spring Kafka documentation states it is deprecated in favor of using the @Transactional annotation, and the S.C.S. example for this problem uses the @Transactional annotation and the default transaction manager created by the start-jpa library (I think)). I can see in my debugger that regardless of whether or not I @EnableTransactionManagement and use a @Transactional on my consumer, the consumer is executed in a kafka transaction using a transaction template higher in the stack, but I do not see a database transaction anywhere.

I have a few questions I want to understand:

• Am I correct that the Kafka Listener Container runs my consumers in the context of a Kafka transaction regardless of whether or not I have a @Transactional annotation? And if so, is there a way to only run specific functions in a Kafka transaction?
• Would the above change for producers, since the container doesn't have a way to intercept calls to the producers (as far as I know)?
• What should I do to synchronize a Kafka and a database transactions so that the DB commit happens before the Kafka commit?

I have the following Crud Repository, collection of handlers, and application.yml:

Trying to configure Spring to send bad messages to dead letter queue while using batch mode. But as a result in dlq topic there is nothing.

I use Spring Boot 2.5.3 and Spring Cloud 2020.0.3. This automatically resolves version of spring-cloud-stream-binder-kafka-parent as 3.1.3.

Here is application.properties:

I use spring boot version 2.5.3, spring-cloud-stream-binder-kafka-stream version 3.1.3 and kafka-clients version 2.8.0. I want to use REPLACE_THREAD option for uncaught exception handler in kafka streams.

But I'm not able to use that since StreamsBuilderFactoryBeanConfigurer (2.6.7 version) doesn't support fb.setUncaughtExceptionHandler(ex -> { log.error("Uncaught exception: ", e); snsService.publish("UncaughtException thrown"); return StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.REPLACE_THREAD; });

Is it possible to replace the streams thread with fb.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler()?

Thanks in Advance!