kafka-mirror-maker | Krux Mirror Maker is a custom solution | Pub Sub library

There are best practices out there which recommend to run the Mirror Maker on the target cluster. https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html

I wonder why this recommendation exists because ultimately all data must cross the border between the clusters, regardless of whether they are consumed at the target or produced at the source. A reason I can imagine is that the Mirror Maker supports multimple consumer but only one producer - so consuming data on the way with the greater latency might be speed up by the use of multiple consumers.

If performance because of multi threading is a point, would it be usefaul to use several producer (one per consumer) to replicate the data (with a custom replication process)? Does anyone knows why the Mirror Maker shares a single Producer among all consumers?

My usecase is the replication of data from several source cluster (~10) to a single target cluster. I would prefer to run the replication process on the source cluster to avoid to many replication processes (each for one source) on the target cluster.

Hints and suggestions on this topic are very welcome.

I have swarm network configured when try run Kafka MirrorMake from service inside a docker stack, I get an error from service logs:

The error described below occurs in MapR 6.0, but I fail to see any MapR-specificity, so guess it could happen in other enrironments..

I'm mirroring a MapR Stream to Azure Kafka-enabled Event Hub, combining these two documents:

Mirroring Topics from a MapR Cluster to an Apache Kafka Cluster

Use Apache Kafka MirrorMaker with Azure Event Hubs for Kafka Ecosystem | Microsoft Docs

There seems to be some activity happening in Azure when I run the mirror making command, but in the end it fails with this exception list:

Basically, MM is replicating MORE than I need it to.

I have four environments, DEV01, DEV02, TST01, and TST02, that each have two Servers running the same App that is generating JSON files. Logstash is reading those files and pushing messages into two, three node Kafka Clusters, KAF01 & KAF02. The DEV01 & TST01 boxes push to the KAF01 Cluster, with corresponding DEV01 & TST01 topics, and the DEV02 & TST02 boxes push to the KAF02 Cluster, with corresponding DEV02 & TST02 topics. Logstash is running on each of the Kafka nodes to then push the messages into corresponding Elasticsearch Clusters. This all works as expected. I then added in MM to replicated messages between environments, IE: DEV01<->DEV02, TST01<->TST02. I started the MM process for the DEV environments and everything worked fine. Then, on the same Hosts, I started a 2nd MM process for the TST environments and everything seemed fine until I realized that I was seeing messages from TST in DEV Elasticsearch and vice versa.

Here's a rough diagram of the flow:

I have MM running on the first Hosts in each Kafka Cluster, IE: kaf01-01 & kaf02-01. For the KAF01 Cluster, kaf01-01 is setup to mirror both the dev01 & tst01 topics to the KAF02 Cluster:

kafka-mirror-maker.sh kafka.tools.MirrorMaker --consumer.config dev01_mm_source.properties --num.streams 1 --producer.config dev01_mm_target.properties --whitelist="dev01"

For --consumer.config, the dev01_mm_source.properties file is configured with the KAF01 Cluster nodes. For --producer.config, the dev01_mm_target.properties file is configured with the KAF02 Cluster nodes.

kafka-mirror-maker.sh kafka.tools.MirrorMaker --consumer.config tst01_mm_source.properties --num.streams 1 --producer.config tst01_mm_target.properties --whitelist="tst01"

For --consumer.config, the tst01_mm_source.properties file is configured with the KAF01 Cluster nodes. For --producer.config, the tst01_mm_target.properties file is configured with the KAF02 Cluster nodes.

For the KAF02 Cluster, kaf02-01 is setup to mirror both the dev02 & tst02 topics to the KAF01 Cluster:

kafka-mirror-maker.sh kafka.tools.MirrorMaker --consumer.config dev02_mm_source.properties --num.streams 1 --producer.config dev02_mm_target.properties --whitelist="dev02"

For --consumer.config, the dev02_mm_source.properties file is configured with the KAF02 Cluster nodes. For --producer.config, the dev02_mm_target.properties file is configured with the KAF01 Cluster nodes.

kafka-mirror-maker.sh kafka.tools.MirrorMaker --consumer.config tst02_mm_source.properties --num.streams 1 --producer.config tst02_mm_target.properties --whitelist="tst02"

For --consumer.config, the tst02_mm_source.properties file is configured with the KAF02 Cluster nodes. For --producer.config, the tst02_mm_target.properties file is configured with the KAF01 Cluster nodes.

Do I have things mixed up? Do I have the --consumer.config and --producer.config files backwards? Is the regex for the --whitelist option that I'm using incorrect? Not really using regex either, just a quoted string. I've triple-checked that Logstash on all of the App boxes is configured to push to the correct Kafka topic and that Logstash on the Kafka boxes is configured to pull from the correct Kafka topic and then push to the correct Elasticsearch Cluster.

Just started working with Kafka and MM today so I'm totally new to all of this and any/all help is greatly appreciated.

I use kafka 0.10.2.0 and Im trying to use the kafka mirror maker option.

consumer.properties:

I am trying to use the mirrormaker tool to replicate data from one primary cluster to backup one, but I got following error.