kafka-connect-spooldir | Kafka Connect connector for reading CSV files into Kafka | Pub Sub library

by jcustenborder Java Version: 1.0.31 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | kafka-connect-spooldir Summary

kafka-connect-spooldir is a Java library typically used in Messaging, Pub Sub, Kafka applications. kafka-connect-spooldir has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. However kafka-connect-spooldir has 10 bugs. You can download it from GitHub, Maven.

Documentation | Confluent Hub. This Kafka Connect connector provides the capability to watch a directory for files and read the data as new files are written to the input directory. Each of the records in the input file will be converted based on the user supplied schema. The connectors in this project handle all different kinds of use cases like ingesting json, csv, tsv, avro, or binary files.

Support

Quality

Security

License

Reuse

Support

kafka-connect-spooldir has a low active ecosystem.

It has 138 star(s) with 117 fork(s). There are 11 watchers for this library.

It had no major release in the last 12 months.

There are 32 open issues and 112 have been closed. On average issues are closed in 139 days. There are 3 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of kafka-connect-spooldir is 1.0.31

Quality

kafka-connect-spooldir has 10 bugs (0 blocker, 0 critical, 3 major, 7 minor) and 134 code smells.

Security

kafka-connect-spooldir has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

kafka-connect-spooldir code analysis shows 0 unresolved vulnerabilities.

There are 2 security hotspots that need review.

License

kafka-connect-spooldir is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kafka-connect-spooldir releases are available to install and integrate.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

It has 4814 lines of code, 281 functions and 76 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed kafka-connect-spooldir and discovered the below as its top functions. This is intended to give you an instant insight into kafka-connect-spooldir implemented functionality, and help decide if they suit your requirements.

Read CSV records
Add record
Return headers from this file
Return the offset of the record
Starts the generator
Generate key schema
Configure parser
Create a LogFieldConverter
Determines the type of the CSV file
Create a CSVarser builder
Poll for a set of source records
Generate a list of task configurations for this connector
Determines the type of the JSON stream
Process input file
Converts a log entry into a date
Create the given directory
Process the batch iterator
Delegate to all input files
Runs a Csv schema generator
Creates a schema for the log field
Process the batch
Compare file attributes
Configure the CSV parser
Reads the JSON parser from the given input file
Convert the Avro data file
Process log entries

Get all kandi verified functions for this library.

kafka-connect-spooldir Key Features

No Key Features are available at this moment for kafka-connect-spooldir.

kafka-connect-spooldir Examples and Code Snippets

No Code Snippets are available at this moment for kafka-connect-spooldir.

Community Discussions

Trending Discussions on kafka-connect-spooldir

How to read text files in a folder and write to Kafka Topic on windows

QUESTION

How to read text files in a folder and write to Kafka Topic on windows

Asked 2021-May-10 at 13:35

I have a locally setup Kafka on my windows 10 machine. Currently I am still learning Kafka and testing its capabilities on my local machine.

I was able to create a topic and run producer and consumer through command prompt successfully using below commands.

...

ANSWER

Answered 2021-May-10 at 13:35

What is the right approach to do this?

Your Python code isn't going to track file offsets or successful reads of a file, so using some other spooldir connector would be best

Also worth pointing out that Kafka records have a default max size of 1MB, and it's generally an anti-pattern to use it for file transfers. Rather, use some shared NAS or blob/object storage for files, and send URI locators through Kafka

read about Spool Dir Connector, but couldn't figure out how to configure it on windows machine

Instructions should be the same as Linux, just different file paths and scripts. Confluent hub cli doesn't have a Windows binary, so you'll need to set it up manually

confluent platform is necessary for configuring Spooldir.

It's not. Besides, Confluent Platform is still using Apache Kafka and Zookeeper, so any errors you would get from it, you should have gotten before

Download the connector

Extract the zip

Find and edit connect-standalone.properties

Edit the plugin.path to include the C:\\path\\to\\kafka-connect-spooldir (make sure this is the parent directory of where the JAR files are)

In the extracted folder, you should have another properties file. Configure that

Once you've configured both properties files, use them to run a standalone connect server

Source https://stackoverflow.com/questions/67401017

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kafka-connect-spooldir

You can download it from GitHub, Maven.
You can use kafka-connect-spooldir like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the kafka-connect-spooldir component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

Contributions are always welcomed! Before you start any development please create an issue and start a discussion. Create a pull request against your newly created issue and we're happy to see if we can merge your pull request. First and foremost any time you're adding code to the code base you need to include test coverage. Make sure that you run mvn clean package before submitting your pull to ensure that all of the tests, checkstyle rules, and the package can be successfully built.

Find more information at: