kafka-connect-file-pulse | multipurpose Kafka Connect connector | Cloud Storage library

by streamthoughts Java Version: v2.12.0 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(4)Vulnerabilities Install Support

kandi X-RAY | kafka-connect-file-pulse Summary

kafka-connect-file-pulse is a Java library typically used in Storage, Cloud Storage, Nodejs, Kafka, Amazon S3 applications. kafka-connect-file-pulse has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub, Maven.

Connect FilePulse is a multipurpose, scalable and reliable, Kafka Connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka. It provides capabilities for reading files from: local-filesystem, Amazon S3, Azure Storage and Google Cloud Storage.

Support

Quality

Security

License

Reuse

Support

kafka-connect-file-pulse has a low active ecosystem.

It has 254 star(s) with 50 fork(s). There are 9 watchers for this library.

It had no major release in the last 12 months.

There are 28 open issues and 242 have been closed. On average issues are closed in 135 days. There are 5 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of kafka-connect-file-pulse is v2.12.0

Quality

kafka-connect-file-pulse has 0 bugs and 0 code smells.

Security

kafka-connect-file-pulse has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

kafka-connect-file-pulse code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

kafka-connect-file-pulse is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kafka-connect-file-pulse releases are available to install and integrate.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

It has 27765 lines of code, 2641 functions and 421 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed kafka-connect-file-pulse and discovered the below as its top functions. This is intended to give you an instant insight into kafka-connect-file-pulse implemented functionality, and help decide if they suit your requirements.

Poll for files
Find the next file input iterator
Returns a new FileInputIterator instance for the given iterator
Adds new file input files
List files to be scheduled
Reads all states to the end of the file
Starts the FilePulseSourceConnector
Return a predicate to create a predicate to apply a file cleanup policy
Start the downloader
Downloads a file from an URL
Lists all the files in the specified bucket
Decompress the given file
Returns the next record
Create a SourceRecord using the provided schema
This method returns an instance of a date time expression function instance
Reads the value of the named property
Evaluates the read expression and writes it to the output stream
Starts the source task
Gets the instance of the function function
Extracts the schema of a type
Evaluates the fields of a typed struct
Generates a list of task configurations
Lists object metadata
Performs an atomic move
Runs the monitor
List objects in a container

Get all kandi verified functions for this library.

kafka-connect-file-pulse Key Features

No Key Features are available at this moment for kafka-connect-file-pulse.

kafka-connect-file-pulse Examples and Code Snippets

No Code Snippets are available at this moment for kafka-connect-file-pulse.

Community Discussions

Trending Discussions on kafka-connect-file-pulse

Kafka connector not behaving as expected

Inputing data in a Kakfa Topic from XML with FilePulse - ExplodeFilter

Kafka connect - file pulse - 'xml attribute' extraction

unable to read avro message via kafka-avro-console-consumer (end goal read it via spark streaming)

QUESTION

Kafka connector not behaving as expected

Asked 2021-Jan-25 at 13:07

I'm attempting to use the Kafka file pulse connector (https://github.com/streamthoughts/kafka-connect-file-pulse) to read data from a file into a Kafka topic.

I start the connector using :

...

ANSWER

Answered 2021-Jan-04 at 13:47

Kafka Connect can be used for streaming data records from files into Kafka using connectors like FilePulse, Spooldir, etc.

But, if your need is to ingest log files (e.g log4j application files) then Logstash or Filebeat can be preferable even if you can use FilePulse for that purpose too.

Regarding your issue, I think the problem come from the config property : file.filter.regex.pattern=test.csv which accepts a regex.

You should use this value: "file.filter.regex.pattern":".*\\.csv$"

Disclaimer: I'm the author of Kafka Connect FilePulse

Source https://stackoverflow.com/questions/65533732

QUESTION

Inputing data in a Kakfa Topic from XML with FilePulse - ExplodeFilter

Asked 2020-Oct-26 at 11:37

I'm creating a kakfa topic which comes from an xml and writes to the topic in avro format. I'm using the file pulse to do this, and in the documentation I saw the ExplodeFilter. I tried to configure according to the documentation, but it is not working. The connect docker console is giving the following error:

...

ANSWER

Answered 2020-Oct-26 at 11:37

The error was due to the ExplodeFilter that did not support dot notation for selecting field. Now, this issue is fixed since Connect FilePulse v1.5.2

https://github.com/streamthoughts/kafka-connect-file-pulse/issues/69

Source https://stackoverflow.com/questions/63711965

QUESTION

Kafka connect - file pulse - 'xml attribute' extraction

Asked 2020-Sep-12 at 05:45

I am trying to use the file pulse connector to read XML file. I am new to Kafka/Kafka Connect/XML processing

For file like below, I'd like to keep the data "unit", and the "string1", "string2". currently, by default the processed payload drop them.

...

ANSWER

Answered 2020-Sep-12 at 05:45

FYI This is now fixed in 1.5.3 version very quickly

Source https://stackoverflow.com/questions/63809219

QUESTION

unable to read avro message via kafka-avro-console-consumer (end goal read it via spark streaming)

Asked 2020-Sep-11 at 12:36

(end goal) before trying out whether i could eventually read avro data, usng spark stream, out of the Confluent Platform like some described here: Integrating Spark Structured Streaming with the Confluent Schema Registry

I'd to verify whether I could use below command to read them:

...

ANSWER

Answered 2020-Sep-10 at 20:11

If you are getting Unknown Magic Byte with the consumer, then the producer didn't use the Confluent AvroSerializer, and might have pushed Avro data that doesn't use the Schema Registry.

Without seeing the Producer code or consuming and inspecting the data in binary format, it is difficult to know which is the case.

The message was produced using confluent connect file-pulse

Did you use value.converter with the AvroConverter class?

Source https://stackoverflow.com/questions/63828704

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kafka-connect-file-pulse

You can download it from GitHub, Maven.
You can use kafka-connect-file-pulse like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the kafka-connect-file-pulse component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .