kafka-cluster | A kafka development cluster using docker | Continuous Deployment library

As far as I know, you can only connect a Kafka Connect worker to one Kafka cluster.

If you have data on different clusters that you want to handle with Kafka Connect then run multiple Kafka Connect worker processes.

A templating tool like Helm will let you fill in the environment-variable values from deploy-time settings. In Helm this would look like:

After you use network it will not be localhost anymore. You need to use your service name as connection.url . Can you try connection.url=http://elasticsearch:9200 and maybe without http

When you mention that you are doing activities on the kafka broker i.e. restarting VMs (these should be controlled restarts of the kafka service and not the containers). What I mean is if you want constant consumption even in the time of maintenance you must consider the following -

• Kafka Brokers must be pulled down for maintenance in a rolling restart fashion, click here for details
• The above is suggested one at a time or depending on the ISR settings in the cluster configs
• Number of partitions and the replication factor must be greater than 1 as if a broker is down for maintenance your topic should not have offline partitions which results in a producer/consumer failure and in turn data loss

A personal suggestion the controller can be pull down as the last one in the rolling restarts to avoid multiple controller switches and reloading of the cluster metadata

When we do a rolling restart for each broker after a maintenance activity, the broker takes some time to come up, i.e. the time taken for it to repopulate the partition metadata and for all the under replicated partitions to return to 0 (This is very important so as not to pressure the controller with multiple restarts, as multiple under replicated partitions amy cause offline/unavailable topic partitions depending on your config)

On top of the above you can definitely tweak the following consumer configs -

• heartbeat.interval.ms - must be lower than session.timeout.ms
• session.timeout.ms
• max.poll.interval.ms THe above can be tweaked based on your connection latencies and kafka cluster status You can read more about them here on the Confluent Docs

It is also possible at time when cluster maintenance activity takes place, the broker assigned to be the partition leader takes time to respond which is significantly greater than the session.timeout.msmax.poll, in the case of which the consumer stops retrying. So tweaking the consumer configs and maintaining sanity in cluster operations is the key to a healthy and continuous kafka integration.*

Note - On a personal opinion, having done cluster upgrades/maintenance activities with over 1Gpbs throughput and we do not face consumption issues (expect a spike in request handler/network handler latencies because of the rebalance). Keeping the above disclaimers in mind with careful execution, updates are easier but definitely time consuming as they are to be executed serially in fashion

More help with documentation for cluster maintenance and consumer behavior tweaks -

• https://docs.confluent.io/current/kafka/post-deployment.html
• Kafka10.1 heartbeat.interval.ms, session.timeout.ms and max.poll.interval.ms
• https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2

This was an issue on my end - I had a typo in the topic name configured for mirroring.

I have been reading this blog post "Kafka Listeners - Explained" and I was able to configure 3 Kafka brokers with the following configuration.

You cannot; you need to disable Boot's auto configuration and configure the infrastructure beans for each cluster yourself.

Boot's auto configuration only supports one cluster.

The Kafka provide various setting to achieve high availability and can be tuned and optimize based on requirement

1. min.insync.replicas: Is the minimum number of copies will be always live at any time to continue running Kafka cluster. e.g. lets we have 3 broker nodes and one broker node got down in that case if min.insync.replicas = 2 or less cluster will keep serving request however if min.insync.replicas 3 it will stop. Please note min.insync.replicas=1 is not advisable in that case if data lost will lost forever. min.insync.replicas is a balance between higher consistency (requiring writes to more than one broker) and higher availability (allowing writes when fewer brokers are available).

2. ack(Acknowledgements): While publishing message we can set how many replica commit before producer receive acknowledge. e.g. ack is 0 means immediately acknowledge the message without waiting any commit to partition. ack is 1 means get success acknowledge after message get commit to the leader. ack is all means acknowledge after all in-sync replicas committed.

leader.election.enable: You can set unclean.leader.election.enable=true on your brokers and in this case, if no replicas are in-sync, one of the out-of-sync replicas will be elected. This can lead to data loss but is favoring availability. Of course if some replicas are in-sync it will still elect one of them

offsets.topic.replication.factor: should be greater than in case __consumer_offsets to have high available. __consumer_offsets is offset topic to manage topic offset so in case you have topic replication factor is 1 it may failed to consumer if one broker got down

__consumer_offsets maintain committed for each topic of consumer group.

Consumer always reads from a leader partition Consumers do not consume from followers, the followers only exist for redundancy and fail-over. There are possibilities that failed broker consists multiple leader partition. So in that case follower from partition on other broker will get promoted to leader. There different scenario lets follower partition doesn't have 100% in-sync with leader partition then we might loss some data. Here scenario comes how or what ack your using while publishing messages.

There is trade off how many number of partition or replication is best number. I can say its depend design and choice based on discussion. Please make note large number of replica will give over burden which doing ISR with all follower partition with more memory occupied whereas very less like 1 or 2 will better performance but not high availability.

Broker controller The controller is one of the brokers that has additional partition and replica management responsibilities. It has some extra responsibility and store some meta details and response in any meta details change in partition.

Any of the brokers can play the role of the controller, but in a healthy cluster there is exactly one controller.Kafka broker will get relected to another controller in case controller shutdown or zookeeper lost connection. If controller is 0 or more than 1 means Broker is not healthy

According to the documentation,

The name of the Kafka topics always takes the form logicalName.databaseName.collectionName, where logicalName is the logical name of the connector as specified with the mongodb.name configuration property, databaseName is the name of the database where the operation occurred, and collectionName is the name of the MongoDB collection in which the affected document existed.

This means that if your connector's logical name is myConnector and your database myDatabase has two collections users and orders