mongo3 | Rule your mongoDB clusters

so as tested with the OP there was a mixup with the network configuration in compose and host network.

when using multiple services in docker-compose each service is a different host in the compose network, so for example if we have multiple services using the same port that won't be an issue because each has a different hostname and IP.

when mapping ports from docker network to host network using bridge adapter usually only one service can use each port and better to use ports >1024 for security and compatibility reasons.

when wanting to connect to different services in the same compose file but no external connection from WAN into the compose network is needed the ports section of the compose file or the -p option in the docker run command isn't needed

after fixing said miss-configuration there shouldn't be any problems around that topic

There are some partial answers on this issue from various places, here is what I think as a complete answer.

• Mongo clients use the hostnames listed in the replica set config, not the seed list

  Although the connection string is "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs", mongo client does not connect to the members of the replica set with seed addresses localhost:27017 etc, instead the client connects to the members in the replica config set returned from the seed hosts, i.e., the ones in the rs.initiate call. This is why the error message is Error connecting to mongo1:27017 instead of Error connecting to localhost:27017.

• Container hostnames are not addressable outside the container network

  A mongo client inside the same container network as the mongo server containers can connect to the server via addresses like mongo1:27017; however, a client on the host, which is outside of the container network, can not resolve mongo1 to an IP. The typical solution for this problem is proxy, see Access docker container from host using containers name for details.

Because the problem involves docker networking and docker networking varies between Linux and Mac. The fixes are different on the two platforms.

The proxy fix (via 3rd party software or modifying /etc/hosts file) works fine but sometimes is not viable, e.g., running on remote CI hosts. A simple self-contained portable solution is to update the intiate_replia_set.sh script to initiate the replica set with member IPs instead of hostnames.

intiate_replia_set.sh

On my system I start the service with sudo systemctl start mongod

You can set everything in the service file, could be like this:

I managed to get it to work, this is a correct configuration...

The message "Unable to connect to the server" was because I had wrongly deployed mongo instance so it's not related to kafka-connect or confluent cloud.

I'm going to leave this question as an example if somebody struggles with this in the future. It took me a while to figure out how to configure docker-compose for kafka-connect that connects to confluent cloud.

Yes, the clients need to be able to resolve the hostnames used to add the nodes to the replica set.

When connecting to a replia set, the hostnames provided in the connection string are used as a seed list.

The client attempts to connect to each node in the seedlist in turn until one succeeds. It then runs the isMaster database command to get the list of hosts in the replica set, and the current primary.

The client drops the initial connection to the seed host and connects to the replica set members using the hostname and port returned in the isMaster command.

From the docs:

isMaster.passives

An array of strings in the format of "[hostname]:[port]" listing all members of the replica set which have a members[n].priority of 0.

This field only appears if there is at least one member with a members[n].priority of 0.

Those nodes have been set to priority 0 somehow, and will therefore never attempt to become primary.

I did multiple research and found that if 2 node shutdown then you can manually made running node as primary.

To have a primary in a 3 node replica set at least 2 nodes must be operational.

The "purpose" is also known as "extended key usage".

Openssl x509v3 Extended Key Usage gives some example code for setting the purposes.

As pointed out by Joe, the documentation states that the certificates must either have no extended key usage at all, or the one in the PEMKeyFile must have server auth, and the one in the cluster file must have client auth.

Finally, I could make the mongo-kafka-connector work on Windows.

Here is what worked for me: Kafka installation folder is E:\Tools\kafka_2.12-2.4.0

E:\Tools\kafka_2.12-2.4.0\plugins has mongo-kafka-1.0.1-all.jar file.

I downloaded this from https://www.confluent.io/hub/mongodb/kafka-connect-mongodb Click on the blue Download button at the left to get mongodb-kafka-connect-mongodb-1.0.1.zip file.

There is also the file MongoSinkConnector.properties in the etc folder inside the zip file. Move it to kafka_installation_folder\plugins

My connect-standalone.properties file has the following entries: