spring-integration-aws | Spring Integration Adapters for Amazon Web | Application Framework library

I have a requirement to consume messages from rabbitMQ, do some processing and finally publish messages to the Kinesis Data stream. We are already using Spring Boot, Spring Integration Core & Spring Integration AQMP 5.5.1 Integration Flows for consuming messages from RabbitMQ. We are not using Spring Cloud Stream for any of our projects.

What spring library do you suggest for the use case to publish messages to the Kinesis data stream? After going through the Spring docs, I see a couple of options available. Could you please advise which is the best one to pursue?

1. spring-cloud-stream-binder-aws-kinesis
2. spring-integration-aws

I am new to spring integration and currently stuck on unit testing my integration flow. My flow looks something like this.

1. Recieve some data from TCP channel adapter in XML format.
2. Convert it to JSON.
3. Send JSON message to amazon sqs queue.

and XML file is :

I have two applications - the first produces messages using spring-cloud-stream/function with the AWS Kinesis Binder, the second is an application that builds off of spring integration to consume messages. Communicating between the two is not a problem - I can send a message from "stream" and handle it easily in "integration".

When I want to send a custom header, then there is an issue. The header arrives at the consumer as an embedded header using the "New" format (Has an 0xff at the beginning, etc.) - See AbstractMessageChannelBinder#serializeAndEmbedHeadersIfApplicable in spring-cloud-stream.

However, the KinesisMessageDrivenChannelAdapter (spring-integration-aws) does not seem to understand the "new" embedded header form. It uses EmbeddedJsonHeadersMessageMapper (See #toMessage) which cannot "decode" the message. It throws a com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'ÿ': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') because of the additional information included in the embedded header (0xff and so on).

I need to send the header across the wire (the header is used to route on the other side), so it's not an option to "turn off" headers on the producer. I don't see a way to use the "old" embedded headers.

I'd like to use spring-cloud-stream/function on the producer side - it's awesome. I wish I could redo the consumer, but...

I could write my own embedded header mapper that understands the new format (use EmbeddedHeaderUtils), and wire it into the KinesisMessageDrivenChannelAdapter.

Given the close relationship between spring-cloud-stream and spring-integration, I must be doing something wrong. Does Spring Integration have an OutboundMessageMapper that understands the new embedded form?

Or is there a way to coerce spring cloud stream to use a different embedding strategy?

I could use Spring Integration on the producer side. (sad face).

Any thoughts? Thanks in advance.

I have int-aws:sqs-message-driven-channel-adapter on which if I set the errorChannel, the downstream exceptions go there.

However, when I don't set an errorChannel, the exception does not get logged. It does not go to the errorChannel which is expected. Is there a way, that such exceptions at least get logged? Is there a default errorlogger which can simply log such errors?

UPDATE

Posting XML and DSL config as per the comments. The error is simulated in the persistence layer by setting null for a @NotBlank field on the ServiceObject.

UPDATE: There is bug in spring-integration-aws-2.3.4

I am integrating SFTP (SftpStreamingMessageSource) as source with S3 as destination. I have similar Spring Integration configuration:

I'm following the docs: Spring Batch Integration combining with the Integration AWS for pooling the AWS S3.

But the batch execution per each file is not working in some situations.

The AWS S3 Pooling is working correctly, so when I put a new file or when I started the application and there's files in the bucket the application sync with the local directory:

I've created an on demand ChannelAdapter, AsyncTaskExecutor and a Channel for every queue registered on the application. I noticed that when the number of maxPoolSize of the AsyncTaskExecutor is equal to one, the messages are not being processed. This is how the AsyncTaskExecutor bean is created.

I want to implement spring-integration-aws to send and receive messages with SQS. I am looking at localstack and would like to know the recommendation of the spring team.

Which tool/api should I use for local setup of spring integration flows for SQS inbound and outbound adapters?

Also, will there be examples of AWS in spring-integration-samples in future? I am looking for an example with xml config that reads the aws config from credentials and send and receive messages via outbound adapters.

When the application stops, kinesis binder tries to unlock dynamoDB and throws unlock failed exception.

I followed this original post with the similar issue and updated spring-integration-aws version to v2.3.1.RELEASE. But still seeing the same error on application shut down.