kafka-exactly-once | Usage examples for the Kafka createDirectStream

My app has 5+ consumers consuming off of five partitions on a kafka topic.(using kafka version 11) My consumer's each produce a message to another topic then save some state to the database, then do a manual_ immediate acknowledgement and move onto the next message.

I'm trying to solve the scenario when they emit successful to the outbound topic. then we have a failure/lose the consumer. When another consumer takes over the partition it will emit ANOTHER message to the outbound topic. This is bad :(

I discovered that kafka now has idempotent producers but from what I read it only guarantees for a producers session.

"When producer restarts, new PID gets assigned. So the idempotency is promised only for a single producer session" - (blog) - https://hevodata.com/blog/kafka-exactly-once

This seems largely useless to me. In my use-case the whole point is when I replay a message on another consumer it does not duplicate the outbound message.

Is there something i'm missing?

I need to apply transactions in a system that comprises of below components:

1. A Kafka producer, this is some external application which would publish messages on a kafka topic.
2. A Kafka consumer, this is a spring boot application where I have configured the kafka listener and after processing the message, it needs to be saved to a NoSQL database.

I have gone through several blogs like this & this, and all of them talks about the transactions in context of streaming application, where the messages would be read-processed-written back to a Kafka topic.

I don't see any clear example or blog around achieving transactionality in the use case similar to mine i.e. producing-processing-writing to a DB in a single atomic transaction. I believe it to be very common scenario & there must be some support for it as well.

Can someone please guide me on how to achieve this? Any relevant code snippet would be greatly appreciated.