kafka-manager | 一站式Apache Kafka集群指标监控与运维管控平台 | Pub Sub library

I am starting my first docker proyect, trying to setup zookeeper and Kafka.

I have the following docker-compose.yml:

I run kafka using docker, have setup storing data on the volume. I setup some source connectors, topics were created automatically with cleanup.policy delete. Using kafka manager I changed policy to compact.

The problem:

After stop/start docker-compose topics are present, but cleanup.policy is reverted back to delete

Question:

How to persist topic configuration after restart?

Additional info

I restart kafka dockers with command:

rm /kafka/data/1/meta.properties; docker-compose down && docker-compose up -d --no-recreate

Docker-compose.yml:

I'm trying to create Kafka cluster automatically, instead of creation manually, I'm using the stable chart: https://github.com/helm/charts/tree/master/stable/kafka-manager

in the template folder there are two .yaml files: configmap.yaml and job.yaml, what's these files and what's the roles of these files?

I have confluent kafka installed running on docker. In the topic i have 10 partitions. The problem is that I cannot consume messages from that topic, but I can Produce messages in the topic. I am trying to consume from the topic using C# confluent.kafka driver 1.5.1 (latest) with librd.kafka 1.5.0 (latest).

The docker-compose file i start kafka with is the following

I am following this tutorial to run Kafka inside a Docker container on windows.

When I try to launch Kafka-Manager by opening http://localhost:9000 in the browser as described there, I get ERR_CONNECTION_REFUSED.

Something I think might be related is that at the first time I ran docker-compose up, PowerShell showed an error saying I needed to run some command first, to open a virtual machine or something like that.

Then I ran the command that PowerShell had told me and then I managed to run docker-compose up successfully. However the tutorial didn't mention anything about it, and since then every time I tried to run docker-compose up I managed to to it without running another command first, even if I closed and reopened PowerShell.

I suspect PowerShell remembers I'm connected to a virtual machine so docker-compose up runs Kafka inside a virtual machine, and therefore I can't reach Kafka-Manager in the browser, although I see shows the following message:

kafkamanager | [info] p.c.s.NettyServer - Listening for HTTP on /0.0.0.0:9000

Edit:

docker logs for kafka container:

I try to write a template, to list the names of my services as well as their external endpoints + ports. However, I don't find any examples or documentation how to select an element from an array, in this case port from the ports array.

I got that far:

In Kafka-manager github page it is written that:

The minimum configuration is the zookeeper hosts which are to be used for kafka manager state. This can be found in the application.conf file in conf directory. The same file will be packaged in the distribution zip file; you may modify settings after unzipping the file on the desired server.

kafka-manager.zkhosts="my.zookeeper.host.com:2181" You can specify multiple zookeeper hosts by comma delimiting them, like so:

kafka-manager.zkhosts="my.zookeeper.host.com:2181,other.zookeeper.host.com:2181" Alternatively, use the environment variable ZK_HOSTS if you don't want to hardcode any values.

ZK_HOSTS="my.zookeeper.host.com:2181"

So my questions are:

1. Does Kafka-manager already contain Zookeeper when I download it?

2. Should I install Zookeeper for Kafka Manager seperately or use already installed Zookeeper used for Apache Kafka?

3. How many Zookeper instances are required for Kafka-Manager?
4. If I should install Zookeeper dedicated to Kafka-Manager, is it okey to install it in the same machine which Kafka-Manager installed or should I create another Zookeeper cluster in different machines?

I wonder what is the best practice?

I setup my kafka containers by follow the tutorial from here: https://success.docker.com/article/getting-started-with-kafka

Then I found that I can't remove the container anymore, even though I tried docker container prune. The containers are still running.

What should I do?

I am attempting to gather kafka topic metrics using JMX so that I can pack them into an object in java. The Kafka Docs show that the MBean requires a client-id, however we do not have any client-ids set up in our system.

What leads me to believe that it is possible to get topic level metrics without a client id is the fact that our instance of the Kafka Manager service is able to pull out topic metrics. I found the MBean they use "kafka.server:name=MessagesInPerSec,topic=topic,type=BrokerTopicMetrics" and attempted to use that with no results.

My current code is