kafka-consumer | Reads data from a Kafka topic and stores to db | Pub Sub library

As the output says, group doesn't exist with --zookeeper

You need to keep your arguments consistent; use --bootstrap-server to list, delete, and describe, assuming your cluster supports this

However, groups delete themselves with no active consumers, so you shouldn't need to run this

All settings are correct, but instead of our class name we have to point to the spring kafka class in custom-interceptors.xml

You'll want to use the --command-config option to set properties files that contain your CCLoud credentials

The very first article neither mentions nor assumes any limitations regarding the number of consumers/producers.

Just put the Samplers for different Kafka instances (or different topics or whatever is your test scenario) under different JMeter Thread Groups and you will be able to concurrently stress multiple endpoints.

If you prefer doing it from different machines - you can run JMeter in distributed mode and point different JMeter slave machines to stress different endpoints using If Controller and __machineName() or __machineIP() functions combination.

Debezium connector is a source connector i.e it is used to read data from postgresSQL into kafka.

If you want to ingest data into postgres sql try using JDBC sink connector

https://docs.confluent.io/kafka-connect-jdbc/current/sink-connector/index.html

As far as Kafka is concerned, the message is consumed as soon as Alpakka Kafka reads it from Kafka.

This is before the actor inside of Alpakka Kafka has emitted it to a downstream consumer for application level processing.

Kafka auto-commit (enable.auto.commit = true) will thus result in the offset being committed before the message has been sent to your actor.

The Kafka docs on offset management do (as of this writing) refer to enable.auto.commit as having an at-least-once semantic, but as noted in my first paragraph, this is an at-least-once delivery semantic, not an at-least-once processing semantic. The latter is an application level concern, and accomplishing that requires delaying the offset commit until processing has completed.

The Alpakka Kafka docs have an involved discussion about at-least-once processing: in this case, at-least-once processing will likely entail introducing manual offset committing and replacing mapAsyncUnordered with mapAsync (since mapAsyncUnordered in conjunction with manual offset committing means that your application can only guarantee that a message from Kafka gets processed at-least-zero times).

In Alpakka Kafka, a broad taxonomy of message processing guarantees:

• hard at-most-once: Consumer.atMostOnceSource - commit after every message before processing
• soft at-most-once: enable.auto.commit = true - "soft" because the commits are actually batched for increased throughput, so this is really "at-most-once, except when it's at-least-once"
• hard at-least-once: manual commit only after all processing has been verified to succeed
• soft at-least-once: manual commit after some processing has been completed (i.e. "at-least-once, except when it's at-most-once")
• exactly-once: not possible in general, but if your processing has the means to dedupe and thus make duplicates idempotent, you can have effectively-once

it seems like you might be using a version of the Framework that does not support msk event definition. It was added in 2.3.0 release: https://github.com/serverless/serverless/blob/master/CHANGELOG.md#230-2020-09-25

The command kafka-consumer-groups.sh --group gonb_cars --describe --bootstrap-server kafka1:6667 will be used to describe one consumer group only, i.e gonb_cars

The command seems correct and would show the correct result, this particular consumer group has only one consumer in it namely consumer-1.

As per the question, you are looking for multiple consumer groups, which can be seen via the -—list parameter.

The rebalance in Kafka is a protocol and is used by various components (Kafka connect, Kafka streams, Schema registry etc.) for various purposes.

In the most simplest form, a rebalance is triggered whenever there is any change in the metadata.

Now, the word metadata can have many meanings - for example:

• In the case of a topic, it's metadata could be the topic partitions and/or replicas and where (which broker) they are stored
• In the case of a consumer group, it could be the number of consumers that are a part of the group and the partitions they are consuming the messages from etc.

The above examples are by no means exhaustive i.e. there is more metadata for topics and consumer groups but I wouldn't go into more details here.

So, if there is any change in:

• The number of partitions or replicas of a topic such as addition, removal or unavailability
• The number of consumers in a consumer group such as addition or removal
• Other similar changes...

A rebalance will be triggered. In the case of consumer group rebalancing, consumer applications need to be robust enough to cater for such scenarios.

So rebalances are a feature. However, in your case it appears that it is happening very frequently so you may need to investigate the logs on your client application and the cluster.

Following are a couple of references that might help:

1. Rebalance protocol - A very good article on medium on this subject
2. Consumer rebalancing - Another post on SO focusing on consumer rebalancing