mongo-kafka | MongoDB Kafka Connector | Pub Sub library

by mongodb Java Version: r1.10.1 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(8)Vulnerabilities Install Support

kandi X-RAY | mongo-kafka Summary

mongo-kafka is a Java library typically used in Messaging, Pub Sub, MongoDB, Kafka applications. mongo-kafka has no vulnerabilities, it has a Permissive License and it has low support. However mongo-kafka has 72 bugs and it build file is not available. You can download it from GitHub, Maven.

The official MongoDB Kafka Connector.

Support

Quality

Security

License

Reuse

Support

mongo-kafka has a low active ecosystem.

It has 293 star(s) with 200 fork(s). There are 38 watchers for this library.

It had no major release in the last 12 months.

mongo-kafka has no issues reported. There are 6 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of mongo-kafka is r1.10.1

Quality

mongo-kafka has 72 bugs (0 blocker, 0 critical, 8 major, 64 minor) and 343 code smells.

Security

mongo-kafka has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

mongo-kafka code analysis shows 0 unresolved vulnerabilities.

There are 6 security hotspots that need review.

License

mongo-kafka is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

mongo-kafka releases are available to install and integrate.

Deployable package is available in Maven.

mongo-kafka has no build file. You will be need to create the build yourself to build the component from source.

Installation instructions are not available. Examples and code snippets are available.

mongo-kafka saves you 8810 person hours of effort in developing the same functionality from scratch.

It has 22924 lines of code, 1199 functions and 215 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed mongo-kafka and discovered the below as its top functions. This is intended to give you an instant insight into mongo-kafka implemented functionality, and help decide if they suit your requirements.

Create a config definition .
Creates a schema from an Avro schema .
Extracts the update description from a change stream .
Validates that a connection can connect to a MongoDB config .
Converts a BsonValue to a SchemaAndValue .
Infers the schema based on the given BsonValue .
Bulk write operations .
Sort records by topic and namespace .
Creates the collection options .
Add server api config .

Get all kandi verified functions for this library.

mongo-kafka Key Features

No Key Features are available at this moment for mongo-kafka.

mongo-kafka Examples and Code Snippets

No Code Snippets are available at this moment for mongo-kafka.

Community Discussions

Trending Discussions on mongo-kafka

How to run the mongo-kafka connector as a source for kafka and integrate that with logstash input to use elasticsearch as a sink?

Specify Kafka Connect connector plugin version

Convert Gradle-Kotlin build to Maven?

Sink kafka topic to mongodb

Kafka connect Developing connector dependencies

ClassNotFoundException: com.mongodb.ConnectionString for Apache Kafka Mongodb connector

MongoDB Kafka Source Connector throws java.lang.IllegalStateException: Queue full when using copy.existing: true

Debezium Kafka connector mongodb : Error connecting kafka connector to mongodb

QUESTION

How to run the mongo-kafka connector as a source for kafka and integrate that with logstash input to use elasticsearch as a sink?

Asked 2020-Dec-26 at 18:52

I have created a build of https://github.com/mongodb/mongo-kafka

But how does this run to connect with my running kafka instance.

Even how stupid this question sound. But there is no documentation seems to be available to make this working with locally running replicaset of mongodb.

All blogs point to using mongo atlas instead.

If you have a good resource, please guide me towards it.

UPDATE 1 --

Used maven plugin - https://search.maven.org/artifact/org.mongodb.kafka/mongo-kafka-connect

Placed it in kafka plugins, restarted kafka.

UPDATE 2 -- How to enable mongodb as source for kafka?

https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSourceConnector.properties

file to be used as a configuration for Kafka

...

ANSWER

Answered 2020-Dec-22 at 19:49

Port 8083 is Kafka Connect, which you start with one of the connect-*.sh scripts.

It is standalone from the broker, and properties do not get set from kafka-server-start

Source https://stackoverflow.com/questions/65404914

QUESTION

Specify Kafka Connect connector plugin version

Asked 2020-Nov-12 at 17:01

How does Kafka deal with multiple versions of the same connector plugin provided in the CLASSPATH? For example, let's say I put both mongo-kafka-1.0.0-all.jar and mongo-kafka-1.1.0-all.jar into the respective directory, in order to allow using both versions, depending on what's needed. Unfortunately, the docs do not give away a way to specify the version of connector.class, I can only assume this is dealt with like classloading is usually dealt with Java-wise.

...

ANSWER

Answered 2020-Nov-12 at 17:01

If you have the same connector plugin that shares the same connector class (e.g. io.confluent.connect.jdbc.JdbcSinkConnector) and you want separate versions of that same connector JAR, you would need to run multiple Kafka Connect workers.

If you have different connectors that use different dependent JARs then this is handled by Kafka Connect's classpath isolation and the plugin.path setting.

Source https://stackoverflow.com/questions/64807729

QUESTION

Convert Gradle-Kotlin build to Maven?

Asked 2020-Sep-09 at 02:13

I need to use a maven build for my project rather than Gradle using Eclipse.

Below is the source code that I will be using: https://github.com/mongodb/mongo-kafka

There are ways to generate pom.xml files using Gradle build (https://www.baeldung.com/gradle-build-to-maven-pom). However, I realized *.kts extension is related to Kotlin DSL rather than groovy. I have used neither of them before.

Is there any possible way to convert this to pom.xml file which can be used for Maven build?

...

ANSWER

Answered 2020-Sep-09 at 02:13

You can't do it automatically if that is what you are asking. While the dependencies section can be converted one to one, the plugins and tasks are gradle specific. You will need to find a matching maven plugin for each one to fulfill the task currently done by the gradle plugins. A better question would be why bother converting? Gradle is perfectly fine to use and eclipse's maven support is historically terrible.

Source https://stackoverflow.com/questions/63803204

QUESTION

Sink kafka topic to mongodb

Asked 2020-Aug-14 at 17:34

I have a project where I need to get data from JSON files using java and sink it into kafka topic, and then sink that data from the topic to mongodb. I have found the kafka-mongodb connector, but the documentation is available only to connect using confluent plateform. I have tried:

Download mongo-kafka-connect-1.2.0.jar from Maven.
Put the file in /kafka/plugins
Added this line "plugin.path=C:\kafka\plugins" in connect-standalone.properties.
created MongoSinkConnector.properties.

...

ANSWER

Answered 2020-Aug-12 at 20:23

You are missing the MongoDB driver. The MongoDB connector jar contains only the classes relevant to Kafka Connect but it still needs a driver to be able to connect to a MongoDB instance. You would need to download that driver and copy the jar file to the same path where you've published your connector ( C:\kafka\plugins ).

To keep things clean you should also create another folder inside that plugins directory ( e.g: C:\kafka\plugins\mongodb ) and move all the stuff relevant to this connector there.

Later Edit:

I went through an old(er) setup that I had with Kafka Connect and MongoDB Sink connector and I found the following jars:

This makes me believe that the kafka-connect-mongdb jar and the mongodb-driver won't be enough. You can give it a try though.

Source https://stackoverflow.com/questions/63330409

QUESTION

Kafka connect Developing connector dependencies

Asked 2020-May-26 at 19:51

To develop my Kafka connector I need to add a connect-API dependency.

Which one I should use?

For example mongodb connector use connect-api from maven central

But links from dev guide go to https://packages.confluent.io/maven/org/apache/kafka/connect-api/5.5.0-ccs/ and beside 5.5.0-ccs there is also 5.5.0-ce version.

So, at this moment last versions are:

2.5.0 from maven central
5.5.0-ccs from packages.confluent.io/maven
5.5.0-ce from packages.confluent.io/maven

What difference between all three variants?

Which one I should use?

...

ANSWER

Answered 2020-May-03 at 11:38

The 5.x version refer to Releases from Confluent whereas the 2.5.0 refers to the Open Source Apache Kafka project. The ccs belongs to the "Confluent Platform" (licensed) and the ce to the community edition of the Confluent Platform. This doc on licenses around Confluent/Kafka will give you more details.

According to Confluent documentation on inter-compatibility you have this relation: Confluent Platform and Apache Kafka Compatibility

Source https://stackoverflow.com/questions/61532385

QUESTION

ClassNotFoundException: com.mongodb.ConnectionString for Apache Kafka Mongodb connector

Asked 2020-Mar-23 at 17:26

I am configuring a Kafka Mongodb sink connector on my Windows machine.

My connect-standalone.properties file has

plugin.path=E:/Tools/kafka_2.12-2.4.0/plugins

My MongoSinkConnector.properties file has

...

ANSWER

Answered 2020-Mar-23 at 17:26

Finally, I could make the mongo-kafka-connector work on Windows.

Here is what worked for me: Kafka installation folder is E:\Tools\kafka_2.12-2.4.0

E:\Tools\kafka_2.12-2.4.0\plugins has mongo-kafka-1.0.1-all.jar file.

I downloaded this from https://www.confluent.io/hub/mongodb/kafka-connect-mongodb Click on the blue Download button at the left to get mongodb-kafka-connect-mongodb-1.0.1.zip file.

There is also the file MongoSinkConnector.properties in the etc folder inside the zip file. Move it to kafka_installation_folder\plugins

My connect-standalone.properties file has the following entries:

Source https://stackoverflow.com/questions/60805921

QUESTION

MongoDB Kafka Source Connector throws java.lang.IllegalStateException: Queue full when using copy.existing: true

Asked 2020-Feb-21 at 09:29

When importing data from mongodb to kafka using the connector, https://github.com/mongodb/mongo-kafka, it throws java.lang.IllegalStateException: Queue full.

I use the default setting of copy.existing.queue.size, which is 16000, and copy.existing: true. What value should I set? The collection size is 10G.

Environment:

...

ANSWER

Answered 2020-Feb-21 at 09:29

Fixed in https://github.com/mongodb/mongo-kafka/commit/7e6bf97742f2ad75cde394d088823b86880cdf4e

and will be released after 1.0.0. So if anyone faces the same issue, please update the version to later than 1.0.0.

Source https://stackoverflow.com/questions/60210757

QUESTION

Debezium Kafka connector mongodb : Error connecting kafka connector to mongodb

Asked 2020-Jan-11 at 12:17

Below are my MongoDB config in /etc/kafka/connect-mongodb-source.properties

...

ANSWER

Answered 2020-Jan-10 at 08:37

First of all, please check the installation of your plugin using the Kafka Connect REST Interface (see details here).

Try to install Kafka Connect plugins using the plugin path mechanism instead of CLASSPATH (more info in the docs).

Source https://stackoverflow.com/questions/59675234

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install mongo-kafka

You can download it from GitHub, Maven.
You can use mongo-kafka like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the mongo-kafka component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .