go-kafka-connect | Go library providing bindings for the Kafka connect API | Pub Sub library

by ricardo-ch Go Version: 2.1.0 License: GPL-3.0

X-Ray Key Features Code Snippets Community Discussions(5)Vulnerabilities Install Support

kandi X-RAY | go-kafka-connect Summary

go-kafka-connect is a Go library typically used in Messaging, Pub Sub, Kafka applications. go-kafka-connect has no bugs, it has no vulnerabilities, it has a Strong Copyleft License and it has low support. You can download it from GitHub.

Go library providing bindings for the Kafka connect API

Support

Quality

Security

License

Reuse

Support

go-kafka-connect has a low active ecosystem.

It has 11 star(s) with 7 fork(s). There are 18 watchers for this library.

It had no major release in the last 12 months.

There are 4 open issues and 11 have been closed. On average issues are closed in 110 days. There are 1 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of go-kafka-connect is 2.1.0

Quality

go-kafka-connect has no bugs reported.

Security

go-kafka-connect has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

go-kafka-connect is licensed under the GPL-3.0 License. This license is Strong Copyleft.

Strong Copyleft licenses enforce sharing, and you can use them when creating open source projects.

Reuse

go-kafka-connect releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed go-kafka-connect and discovered the below as its top functions. This is intended to give you an instant insight into go-kafka-connect implemented functionality, and help decide if they suit your requirements.

getCreateCmdConfig retrieves the configuration from the command line
IsUpToDate checks if connector is up to date
getUpdateCmdConfig returns the config for the update command
tryUntil runs the given time until the given limit is reached .
getClient returns client instance
getConfigFromFolder returns the list of create connectors .
handleCmd handles configuration
RunEUpdate updates an existing connector
RunECreate executes the connector
newBaseClient returns a new BaseClient .

Get all kandi verified functions for this library.

go-kafka-connect Key Features

No Key Features are available at this moment for go-kafka-connect.

go-kafka-connect Examples and Code Snippets

No Code Snippets are available at this moment for go-kafka-connect.

Community Discussions

Trending Discussions on go-kafka-connect

How to run the mongo-kafka connector as a source for kafka and integrate that with logstash input to use elasticsearch as a sink?

Sink kafka topic to mongodb

ClassNotFoundException: com.mongodb.ConnectionString for Apache Kafka Mongodb connector

MongoDB Kafka Source Connector throws java.lang.IllegalStateException: Queue full when using copy.existing: true

How to stream data from Kafka to MongoDB by Kafka Connector

QUESTION

How to run the mongo-kafka connector as a source for kafka and integrate that with logstash input to use elasticsearch as a sink?

Asked 2020-Dec-26 at 18:52

I have created a build of https://github.com/mongodb/mongo-kafka

But how does this run to connect with my running kafka instance.

Even how stupid this question sound. But there is no documentation seems to be available to make this working with locally running replicaset of mongodb.

All blogs point to using mongo atlas instead.

If you have a good resource, please guide me towards it.

UPDATE 1 --

Used maven plugin - https://search.maven.org/artifact/org.mongodb.kafka/mongo-kafka-connect

Placed it in kafka plugins, restarted kafka.

UPDATE 2 -- How to enable mongodb as source for kafka?

https://github.com/mongodb/mongo-kafka/blob/master/config/MongoSourceConnector.properties

file to be used as a configuration for Kafka

...

ANSWER

Answered 2020-Dec-22 at 19:49

Port 8083 is Kafka Connect, which you start with one of the connect-*.sh scripts.

It is standalone from the broker, and properties do not get set from kafka-server-start

Source https://stackoverflow.com/questions/65404914

QUESTION

Sink kafka topic to mongodb

Asked 2020-Aug-14 at 17:34

I have a project where I need to get data from JSON files using java and sink it into kafka topic, and then sink that data from the topic to mongodb. I have found the kafka-mongodb connector, but the documentation is available only to connect using confluent plateform. I have tried:

Download mongo-kafka-connect-1.2.0.jar from Maven.
Put the file in /kafka/plugins
Added this line "plugin.path=C:\kafka\plugins" in connect-standalone.properties.
created MongoSinkConnector.properties.

...

ANSWER

Answered 2020-Aug-12 at 20:23

You are missing the MongoDB driver. The MongoDB connector jar contains only the classes relevant to Kafka Connect but it still needs a driver to be able to connect to a MongoDB instance. You would need to download that driver and copy the jar file to the same path where you've published your connector ( C:\kafka\plugins ).

To keep things clean you should also create another folder inside that plugins directory ( e.g: C:\kafka\plugins\mongodb ) and move all the stuff relevant to this connector there.

Later Edit:

I went through an old(er) setup that I had with Kafka Connect and MongoDB Sink connector and I found the following jars:

This makes me believe that the kafka-connect-mongdb jar and the mongodb-driver won't be enough. You can give it a try though.

Source https://stackoverflow.com/questions/63330409

QUESTION

ClassNotFoundException: com.mongodb.ConnectionString for Apache Kafka Mongodb connector

Asked 2020-Mar-23 at 17:26

I am configuring a Kafka Mongodb sink connector on my Windows machine.

My connect-standalone.properties file has

plugin.path=E:/Tools/kafka_2.12-2.4.0/plugins

My MongoSinkConnector.properties file has

...

ANSWER

Answered 2020-Mar-23 at 17:26

Finally, I could make the mongo-kafka-connector work on Windows.

Here is what worked for me: Kafka installation folder is E:\Tools\kafka_2.12-2.4.0

E:\Tools\kafka_2.12-2.4.0\plugins has mongo-kafka-1.0.1-all.jar file.

I downloaded this from https://www.confluent.io/hub/mongodb/kafka-connect-mongodb Click on the blue Download button at the left to get mongodb-kafka-connect-mongodb-1.0.1.zip file.

There is also the file MongoSinkConnector.properties in the etc folder inside the zip file. Move it to kafka_installation_folder\plugins

My connect-standalone.properties file has the following entries:

Source https://stackoverflow.com/questions/60805921

QUESTION

MongoDB Kafka Source Connector throws java.lang.IllegalStateException: Queue full when using copy.existing: true

Asked 2020-Feb-21 at 09:29

When importing data from mongodb to kafka using the connector, https://github.com/mongodb/mongo-kafka, it throws java.lang.IllegalStateException: Queue full.

I use the default setting of copy.existing.queue.size, which is 16000, and copy.existing: true. What value should I set? The collection size is 10G.

Environment:

...

ANSWER

Answered 2020-Feb-21 at 09:29

Fixed in https://github.com/mongodb/mongo-kafka/commit/7e6bf97742f2ad75cde394d088823b86880cdf4e

and will be released after 1.0.0. So if anyone faces the same issue, please update the version to later than 1.0.0.

Source https://stackoverflow.com/questions/60210757

QUESTION

How to stream data from Kafka to MongoDB by Kafka Connector

Asked 2019-Jul-09 at 07:07

I want to stream data from Kafka to MongoDB by using Kafka Connector. I found this one https://github.com/hpgrahsl/kafka-connect-mongodb. But there is no step to do.

After googling, it seems to lead to Confluent Platform what I don't want to use.

Could anyone share me document/guideline how to use kafka-connect-mongodb without using Confluent Platform or another Kafka Connector to stream data from Kafka to MongoDB?

Thank you in advance.

What I tried

Step1: I download mongo-kafka-connect-0.1-all.jar from maven central

Step2: copy jar file to a new folder plugins inside kafka (I use Kafka on Windows, so the directory is D:\git\1.libraries\kafka_2.12-2.2.0\plugins)

Step3: Edit file connect-standalone.properties by adding a new line plugin.path=/git/1.libraries/kafka_2.12-2.2.0/plugins

Step4: I add new config file for mongoDB sink MongoSinkConnector.properties

...

ANSWER

Answered 2019-Jul-04 at 10:59

There is an official source and sink connector from MongoDB themselves. It is available on Confluent Hub: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

If you don't want to use Confluent Platform you can deploy Apache Kafka yourself - it includes Kafka Connect already. Which plugins (connectors) you use with it is up to you. In this case you would be using Kafka Connect (part of Apache Kafka) plus kafka-connect-mongodb (provided by MongoDB).

Documentation on how to use it is here: https://github.com/mongodb/mongo-kafka/blob/master/docs/sink.md

Source https://stackoverflow.com/questions/56880527

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install go-kafka-connect

run go get -u github.com/ricardo-ch/go-kafka-connect then inside repo run: make install to install dependencies.
Go 1.9
Docker (for testing purpose only)

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: