kafka-go | Kafka library in Go | Pub Sub library

I need to wrap kcat in a Go function to read a series of topic messages, so thought to use exec.Command() for this is as follows:

I ran into a repo that seems to be a Go module, but there's no go.mod file in it: github.com/confluentinc/confluent-kafka-go.

Is it ok for a go module to have no go.mod file with dependencies, or the authors of that library just didn't migrate to modules yet?

I saw related problems in stackoverflow, they advised to use the interface. But I don't know how to integrate it into my code...

I have main function in

project.com/exec

I'm wondering if there's any logic which determines which consumer reads a message within the same consumer group. I have a single topic, and a single consumer group. However, I have one or more consumers because there's a consumer deployed in production environment and when I run my application locally another consumer is created which subscribes to the same topic (it's a test project so it's not real production and I'm not worried about loss of data). I noticed that interestingly always the local consumer consumes any given message. So it looks like that the consumer which was created later takes precedence.

Is it possible to configure kafka such that a consumer which was created earlier takes precedence for reads?

My setup includes 3 brokers and 1 consumer group id. In addition this property auto.offset.reset is set to earliest (changin it to latest doesn't resolve the issue). I'm using this Go library for kafka. This is my setup code:

Can we configure librdkafka and confluet-kafka-go to use jks files

I'm going to be switching from RabbitMQ to Kafka. This is just a simple spike to see how Kafka operates. I'm not sure if there are settings that I am missing, if it is my code, if it is Kafka-Go, or if this is expected Kafka behavior.

I've tried adjusting the BatchSize as well as the BatchTimeout but neither have had an impact.

The code below creates a topic with 6 partitions and a replication factor of 3. It then produces an incrementing message every 100ms. It launches 6 consumers, one for each partition. Both reading and writing are performed in go routines.

In the log below, it goes 7 seconds without receiving a message and then receives bursts. I'm using Confluent's platform so I recognize that there will be some network latency but not to the degree that I'm seeing.

My basic understanding of a goroutine is that its a simplified way to create a thread.

Looking at the confluent-kafka-go library, the following code is given as an example:

I can't build this simple example of confluent kafka using Docker. Probably a trick with go path or an special build parameter, can't find out, tried all the default folders from go, no success.

Dockerfile

I have a go routine which is basically acting as a KafkaConsumer, it reads messages from a topic and then spawns another go routine for each message it receives. Now this Consumer go routine is supposed to be shutdown when the application which is the main go routine shuts down. But I am facing difficulties in properly shutting this down. Below is the Kafka Consumer definition