kafka-connect-ftp | Kafka Connect Source for FTP server | Pub Sub library

by Eneco Scala Version: 0.1.4 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | kafka-connect-ftp Summary

kafka-connect-ftp is a Scala library typically used in Messaging, Pub Sub, Kafka applications. kafka-connect-ftp has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Monitors files on an FTP server and feeds changes into Kafka. Remote directories of interest are to be provided. On a specified interval, the list of files in the directories is refreshed. Files are downloaded when they were not known before, or when their timestamp or size are changed. Only files with a timestamp younger than the specified maximum age are considered. Hashes of the files are maintained and used to check for content changes. Changed files are then fed into Kafka, either as a whole (update) or only the appended part (tail), depending on the configuration. Optionally, file bodies can be transformed through a pluggable system prior to putting it into Kafka.

Support

Quality

Security

License

Reuse

Support

kafka-connect-ftp has a low active ecosystem.

It has 41 star(s) with 24 fork(s). There are 17 watchers for this library.

It had no major release in the last 12 months.

There are 12 open issues and 13 have been closed. On average issues are closed in 32 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of kafka-connect-ftp is 0.1.4

Quality

kafka-connect-ftp has 0 bugs and 1 code smells.

Security

kafka-connect-ftp has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

kafka-connect-ftp code analysis shows 0 unresolved vulnerabilities.

There are 1 security hotspots that need review.

License

kafka-connect-ftp is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kafka-connect-ftp releases are not available. You will need to build from source code and install.

Installation instructions are not available. Examples and code snippets are available.

It has 775 lines of code, 75 functions and 14 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of kafka-connect-ftp

Get all kandi verified functions for this library.

kafka-connect-ftp Key Features

No Key Features are available at this moment for kafka-connect-ftp.

kafka-connect-ftp Examples and Code Snippets

No Code Snippets are available at this moment for kafka-connect-ftp.

Community Discussions

Trending Discussions on kafka-connect-ftp

Kafka Connect/Landoop: A Recursive Path for FtpSourceConfig

Kafka connect ftp

QUESTION

Kafka Connect/Landoop: A Recursive Path for FtpSourceConfig

Asked 2019-Sep-11 at 07:03

I am using Kafka connect with the kafka-connect-ftp plugin by Landoop (https://docs.lenses.io/connectors/source/ftp.html .)

Is there a way to use connect.ftp.monitor.tail with a recursive path definition, in order to track all files in a directory recursively?

I've tried to do something like

...

ANSWER

Answered 2019-Sep-11 at 07:03

Looks like the code only supports single star glob patterns, and matches on filenames, not directories

https://github.com/Landoop/stream-reactor/blob/master/kafka-connect-ftp/src/main/scala/com/datamountaineer/streamreactor/connect/ftp/source/FtpFileLister.scala#L41

Sounds like a good idea for a github issue, though

Source https://stackoverflow.com/questions/57883139

QUESTION

Kafka connect ftp

Asked 2019-Jun-03 at 23:21

While running the connectors for kafka-connect-ftp showing this below error

...

ANSWER

Answered 2018-Jan-16 at 19:01

There are two main issues with this connector:

It bundles, along with its dependencies, classes from Connect's API. E.g. classes in the package org.apache.kafka.connect. This is not advised and such dependencies should be marked as provided.
The actual reason that the connector fails with classloading issues is that it is depending (at least according to its master branch) on a version of Apache Kafka that probably does not match the version of the deployed Connect worker. Specifically, it depends on kafkaVersion = '0.10.2.0' which is not the latest. Kafka Connect, in its recent versions that offer classloading isolation, will ignore what it considers system classes, such as classes in org.apache.kafka.connect when they are imported by connectors' jars. Instead, it will load such classes from the Kafka Connect jars that ship with Apache Kafka.

The above issues may cause classloading failures, as the ones you observe.

Ideally, they should be addressed at the connector level.

Workarounds you may apply are:

build the connector code from source, after upgrading the Kafka version it depends upon and also marking its Kafka dependencies as provided (including Kafka Connect and Kafka Clients dependencies). Or,
downgrade your deployed Kafka Connect version to match exactly the version that the connector currently depends upon.

A similar issue has been recorded here: kafka mongodb sink connector not starting

Source https://stackoverflow.com/questions/48276945

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kafka-connect-ftp

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: