kafka-docker | Dockerfile for Apache Kafka | Pub Sub library

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR changed from 1 to 3.

The offsets topic only gets created on a brand new cluster. It will not retroactively get updated if you modify offsets.topic.replication.factor.

new topics created OK with replication factor 3

Correct, however, any topic created before you increased default.replication.factor will also only have one replica.

Reassign partition tool doesn't take into account your server properties. You either need a different tool or manually place the replicas yourself. - How to change the number of replicas of a Kafka topic?

A few things from first glance:

Exposed ports and advertised listeners

You are exposing port 29092 on your docker-compose which is configured to the "kafka" host name. You will likely not be able to communicate with Kafka from your host machine this way if you wanted to. You need to expose 9092:9092 (i.e. localhost) so the destination name from the client (localhost) matches the host name on the port if that makes sense. That compose file you've linked exposes 29092, but correctly uses 29092 on "localhost" and 9092 on "kafka" to make it work, you've set it the other way around.

Container-container communication via container name

Your job function is running on a self-contained flink container, trying to connect to bootstrap-server "localhost:9092", which won't work as it's trying to access itself. You need to change that to "kafka:29092" so it communicates with the Kafka container. If you are running the flink job on your host machine, localhost:9092 would work.

Other stuff

Make those changes and see if you have luck. You should be able to see the broker server from your host machine via localhost:9092 (use a Kafka client, portqry, netcat etc.) and from your flink containers via kafka:29092 (likely use netcat etc.).

If there are networking issues with the bridge, try adding your flink containers to the same docker-compose file and remove the "networks" entry from the YAML file to start the group by default on their own network. Providing they are on their own network and configured properly, containers can communicate via container name.

Kafka listeners basically need to match what the client connects to. So if you connect from your host machine to "localhost:9092", your listener for 9092 needs to be "localhost" so the expected names match, otherwise connection will fail. If we connect from another container to port 29092 via hostname "kafka", that listener needs to be configured to "kafka:29092", and so on.

If none of that works, post some logs of the Kafka server container and we'll go from there.

If you are running your consuming .NET app outside of Docker, you should try to connect to localhost:9092. The kafka hostname is only valid in Docker.

You'll find an example of how to run Kafka in Docker and consume the messages from it using a .NET Core app here.

You could compare the docker-compose.yml from that example with yours.

Here is how the the .NET Core app sets up the consumer:

With a user-defined bridge network, containers can resolve each other by name or alias. Thus, in your case the application.yml should point at the kafka container name:

If your container has port forwarded, like "27017:27017" then you can connect it with any mongo client sitting on your machine.

Example: mongo -u -p 127.0.0.1/