mongo-kafka | send mongo oplog stream to kafka | Pub Sub library

If you have the same connector plugin that shares the same connector class (e.g. io.confluent.connect.jdbc.JdbcSinkConnector) and you want separate versions of that same connector JAR, you would need to run multiple Kafka Connect workers.

If you have different connectors that use different dependent JARs then this is handled by Kafka Connect's classpath isolation and the plugin.path setting.

You can't do it automatically if that is what you are asking. While the dependencies section can be converted one to one, the plugins and tasks are gradle specific. You will need to find a matching maven plugin for each one to fulfill the task currently done by the gradle plugins. A better question would be why bother converting? Gradle is perfectly fine to use and eclipse's maven support is historically terrible.

You are missing the MongoDB driver. The MongoDB connector jar contains only the classes relevant to Kafka Connect but it still needs a driver to be able to connect to a MongoDB instance. You would need to download that driver and copy the jar file to the same path where you've published your connector ( C:\kafka\plugins ).

To keep things clean you should also create another folder inside that plugins directory ( e.g: C:\kafka\plugins\mongodb ) and move all the stuff relevant to this connector there.

Later Edit:

I went through an old(er) setup that I had with Kafka Connect and MongoDB Sink connector and I found the following jars:

This makes me believe that the kafka-connect-mongdb jar and the mongodb-driver won't be enough. You can give it a try though.

The 5.x version refer to Releases from Confluent whereas the 2.5.0 refers to the Open Source Apache Kafka project. The ccs belongs to the "Confluent Platform" (licensed) and the ce to the community edition of the Confluent Platform. This doc on licenses around Confluent/Kafka will give you more details.

According to Confluent documentation on inter-compatibility you have this relation: Confluent Platform and Apache Kafka Compatibility

Finally, I could make the mongo-kafka-connector work on Windows.

Here is what worked for me: Kafka installation folder is E:\Tools\kafka_2.12-2.4.0

E:\Tools\kafka_2.12-2.4.0\plugins has mongo-kafka-1.0.1-all.jar file.

I downloaded this from https://www.confluent.io/hub/mongodb/kafka-connect-mongodb Click on the blue Download button at the left to get mongodb-kafka-connect-mongodb-1.0.1.zip file.

There is also the file MongoSinkConnector.properties in the etc folder inside the zip file. Move it to kafka_installation_folder\plugins

My connect-standalone.properties file has the following entries:

Fixed in https://github.com/mongodb/mongo-kafka/commit/7e6bf97742f2ad75cde394d088823b86880cdf4e

and will be released after 1.0.0. So if anyone faces the same issue, please update the version to later than 1.0.0.

First of all, please check the installation of your plugin using the Kafka Connect REST Interface (see details here).

Try to install Kafka Connect plugins using the plugin path mechanism instead of CLASSPATH (more info in the docs).

Mongo-source doesn't support tasks.max > 1. Even if you set it greater than 1 only one task will be pulling data from mongo to Kafka.

How many task is created depends on particular connector. Function List> Connector::taskConfigs(int maxTasks), (that should be overridden during the implementation of your connector) return the list, which size determine number of Tasks. If you check mongo-kafka source connector you will see, that it is singletonList.

https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/MongoSourceConnector.java#L47

I figured it out.

https://docs.confluent.io/current/connect/transforms/timestampconverter.html