spring-cloud-stream | Framework for building Event-Driven Microservices | Reactive Programming library

I have an application with multiple suppliers. So, I'm trying to configure fixed-delay for the specific supplier in Spring Cloud Stream. Example:

application.yaml

I am new to spring framework. I have a confusion regarding spring boot and spring cloud.

I used https://start.spring.io/ to initialize a spring boot application. I think I am using the spring boot framework. However, I would like to use some spring cloud dependencies such as spring-cloud-stream-binder-kafka.

Question 1: If I added this dependency above to my spring boot application, I am wondering if I still can go with the spring boot framework, or I have to change to spring cloud framework.

Question 2: I am wondering if there is any difference when deploying the spring boot or spring cloud application. Or, they just have the different frameworks, and we could deploy them in the same way.

Thank you so much!

I have two applications - the first produces messages using spring-cloud-stream/function with the AWS Kinesis Binder, the second is an application that builds off of spring integration to consume messages. Communicating between the two is not a problem - I can send a message from "stream" and handle it easily in "integration".

When I want to send a custom header, then there is an issue. The header arrives at the consumer as an embedded header using the "New" format (Has an 0xff at the beginning, etc.) - See AbstractMessageChannelBinder#serializeAndEmbedHeadersIfApplicable in spring-cloud-stream.

However, the KinesisMessageDrivenChannelAdapter (spring-integration-aws) does not seem to understand the "new" embedded header form. It uses EmbeddedJsonHeadersMessageMapper (See #toMessage) which cannot "decode" the message. It throws a com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ÿ': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') because of the additional information included in the embedded header (0xff and so on).

I need to send the header across the wire (the header is used to route on the other side), so it's not an option to "turn off" headers on the producer. I don't see a way to use the "old" embedded headers.

I'd like to use spring-cloud-stream/function on the producer side - it's awesome. I wish I could redo the consumer, but...

I could write my own embedded header mapper that understands the new format (use EmbeddedHeaderUtils), and wire it into the KinesisMessageDrivenChannelAdapter.

Given the close relationship between spring-cloud-stream and spring-integration, I must be doing something wrong. Does Spring Integration have an OutboundMessageMapper that understands the new embedded form?

Or is there a way to coerce spring cloud stream to use a different embedding strategy?

I could use Spring Integration on the producer side. (sad face).

Any thoughts? Thanks in advance.

I am developping a simple spring boot application using spring cloud stream and kafka.

I get this error when I added kafka consumer bean.

Spring boot version: 2.5.0

Spring cloud version: 2020.0.3-SNAPSHOT

Kafka client version: 2.7.1

Error log:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder.createConsumerEndpoint(KafkaMessageChannelBinder.java:716)

The following method did not exist:

org.springframework.kafka.listener.ContainerProperties.setAckOnError(Z)V

pom.xml file:

I am new to Kafka Streams and Spring Cloud Stream but have read good things about it in terms of moving the integration related codes into properties file so devs can focus mostly on the business logic side of things.

Here I have my simple application class.

I am trying to create junit tests an application that makes use of Spring Cloud Stream to recieve events from RabbitMQ. In other application I generally import spring-cloud-stream-test-support and spring-rabbit-test and annotate the test class with @SpringBootTest and we're good to go. However, @SpringBootTest loads the entire application context which is not ideal in this application as it is quite large and would require the mocking of too many beans which are irrlevant to the test. Therefore, I tried limiting the context by specifying the classes I want loaded as follows: @SpringBootTest(classes = {MessageProcessor.class, Consumer.class}). It seems like this is not enough as I'm getting a Dispatcher has no subscribers for channel error.

So my question is, what are the minimum classes that need to be included in the context to test SpringCloudStream/RabbitMQ consumer?

I am trying to deduplicate the records using input topic as KTable and sinking them to output topic. But the KTable is still sinking the duplicate records to the output topic. Not sure where am I going wrong.

Here is my application.yml

For example, I configure a topic have 2 partitions, but in my application with 1 instance, I use Flux.parallel(10) to consume the message, and it has 1000 message lag on that topic, what will happen?

1. Will it poll 10 messages per time? from 2 partitions or 1 partition?
2. Only poll 2 message and 1 partition each?

I want to know how it works, so I could configure it right with the capability of large throughput and consume sequence

BTW I found this issue, but now answer there

I'm trying to use one transaction manager (ChainedTransactionManager) for Rabbit and Kafka, chaining RabbitTransactionManager and KafkaTransactionManager. We intend to achieve a Best effort 1-phase commit.

To test it, the transactional method throws an exception after the 2 operations (sending a message to a Rabbit exchange and publishing and event in Kafka). When running the test, the logs suggest a rollback is initiated but the message ends up in Rabbit anyway.

• Notes:

• We're using QPid to simulate in-memory RabbitMQ for testing (version 7.1.12)
• We're using an in-memory Kafka for testing (spring-kafka-test)
• Other relevant frameworks/libraries: spring-cloud-stream

Here's the method where the problem occurs: