zookeeper.client | An implementation of ZooKeeper client | Runtime Evironment library

You need to change back

If data goes to one broker, then the other, that depends on your partition count and excludes any connection issues

Still wonder is it a cluster now ? , however both the nodes have their own zookeepers

First, as mentioned in the documentation, zookeeper.connect needs to be the same string for a cluster to be formed. You only need one Zookeeper server, so stop the one on 2180 and just use localhost:2181

Once both Kafka brokers are running, you can use kafkacat -L or in Java AdminClient.describeCluster to verify the cluster metadata

As I mentioned in the question my diagnosis revealed that "InconsistentClusterIdException" is the culprit in this situation. So somewhere a cluster.id is defined which is having an older value, and new cluster.id is not the same as the one given in the files defined in the system.

The log also mentions that the file where the cluster.id is defined is named "meta.properties", now the question is where to find this file.

I checked the kafka installation directory there is a logs folder, first I thought the meta.properties would be there but alas! that was not to be the case.

Then I ran the following command at the root directory to find the meta.properties file location.

If a zookeeper quorum is maintained during the upgrade, then the Flink job manager should not be impacted. Otherwise it's not surprising that the job manager would fail.

Normally you would upgrade the zookeeper followers first, one by one, and then upgrade the leader last. Verify that the quorum has been reestablished before taking down another node.

I have this working now. It appears to be because I repeatedly brought the cluster down and up and didn't properly clear the networking state, which probably led to some sort of black-holing somewhere.

It may be overkill, but what I ended up doing was simply flushing the iptables rules and restarting all relevant services like docker which required special iptables rules. Now that the cluster is working, I don't envision repeatedly re-creating the cluster.

future.cancel(true);

This is interrupting the producer thread and is likely the root cause of the problem.

You should use future.cancel(false); to allow the task to terminate in an orderly fashion, without interruption.

Yes, you do have to rebuild your Flink jobs whenever you update the Flink version being used to run them. The libraries you use should be from the same exact version used by the Job Manager and Task Managers.

If you are trying to automate deployments for a CI/CD pipeline, you could inject the version number into the pom.xml using an environment variable -- but doing things like that can make it hard to debug when things go wrong.

Many Zookeeper instances were running earlier. I killed all the zookeeper and Brokers , restarted them again freshly . It is working fine now.

this should work !