kafka | Load-balancing resuming Kafka consumer for go | Pub Sub library

I have a Spring Boot app with a Kafka Listener implementing the BatchAcknowledgingMessageListener interface. When I receive what should be a single message from the topic, it's actually one message for each line in the original message, and I can't cast the message to a ConsumerRecord.

The code producing the record looks like this:

I am trying to figure out is there any way to send failed records in Dead Letter topic in Spring Boot Kafka in Batch mode. I don't want to make the records being sent in duplicate as it's consuming in batch and few are already processed. I saw this link ofspring-kafka consumer batch error handling with spring boot version 2.3.7

I thought about a use case to stop container and start again without using DLT but again the issue of duplication will come in Batch mode.

@Garry Russel can you please provide a small code for batch error handling.

How can I execute the below in a transaction. My requirement is message offset should not be committed to Kafka if the DB calls fails .Kafka consumer configuration is here https://pastebin.com/kq5S9Jrx

This probably ins't typical setup, but due to higher decisions we endup having multiple kafka clusters within one app, multiple topics per each, and each might have different serializing strategy. Json/avro. And avro might be with confluent schema registry or using single object encoding.

Well I got it working somehow, by building my own abstractions and registry which analyzes the configuration and creates most of stuff manually, but I feel I needed to repeat stuff like topic names, schema registry url on several places multiple times just to create all needed beans. Ugly as hell.

I'd like to ask, if there is some better way and support for this I just might have overlooked.

I need to create N representations of kafka clusters, configuring it once. Configure topics respective to given kafka cluster, configure confluent schema registry for topics where applicable etc, so that I can create instance of Avro schema file, send it to KafkaTemplate and it will work.

I need a way to force the compaction of the __consumer_offsets topic. In a test environment I tried to delete the file cleaner-offset-checkpoint and then kafka deleted many segments as you can see below. Is it safe to delete this file in a production environment?

Before removing cleaner-offset-checkpoint:

We want to replace the path on /etc/fstab file from

I followed the instructions at Structured Streaming + Kafka and built a program that receives data streams sent from kafka as input, when I receive the data stream I want to pass it to SparkSession variable to do some query work with Spark SQL, so I extend the ForeachWriter class again as follows:

I am using the SQL connector to capture CDC on a table that we only expose a subset of all columns on the table. The table has two unique indexes A & B on it. Neither index is marked as the PRIMARY INDEX but index A is logically the primary key in our product and what I want to use with the connector. Index B references a column we don't expose to CDC. Index B isn't truly used in our product as a unique key for the table and it is only marked UNIQUE as it is known to be unique and marking it gives us a performance benefit.

This seems to be resulting in the error below. I've tried using the message.key.columns option on the connector to specify index A as the key for this table and hopefully ignore index B. However, the connector seems to still want to do something with index B

1. How can I work around this situation?
2. For my own understanding, why does the connector care about indexes that reference columns not exposed by CDC?
3. For my own understanding, why does the connector care about any index besides what is configured on the CDC table i.e. see CDC.change_tables.index_name documentation

spring-kafka creates a ValueSerializer instance in the AbstractConfig class using a no-args constructor.

I can see that JsonSerializer has an ObjectMapper constructor which I would like to use to inject a preconfigured ObjectMapper bean.

The default ObjectMapper includes null values in the response which I would like to remove. I added spring.jackson.default-property-inclusion: NON_EMPTY to my properties.yml but since Spring creates a default instance, this does not help me.

Could someone point me in the right direction?