flume-kafka | A kafka source & sink for flume | Stream Processing library

by frankyaorenjie Java Version: Current License: No License

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | flume-kafka Summary

flume-kafka is a Java library typically used in Data Processing, Stream Processing, Kafka, Hadoop applications. flume-kafka has no bugs, it has no vulnerabilities, it has build file available and it has low support. You can download it from GitHub.

Since this plugin for flume is going to merge into flume, I’ve splited this plugin to two dependent plugin [flumg-ng-kafka-source] and [flume-ng-kafka-sink] ASFv2 branch is okay, but I advised to use new plugin. This project is used for [flume-ng] to communicate with [kafka 0.7,2] For v0.2 now, I think the parameters pass to flume-kafka need to be handled by users, not by code. Before this version, I add many parameters of kafka and their default value in code. That is to say, whatever parameters you write in conf file, they will be passed to Kafka producer or consumers. I cannot control if the parameters you wrote will take effect. The responsibilites for using correct parameters or find out what parameters to use, in my opinion, are yours. On the other hand, it is simple if Kafka add some new parameters:). Configuration of Kafka Sink.

Support

Quality

Security

License

Reuse

Support

flume-kafka has a low active ecosystem.

It has 72 star(s) with 52 fork(s). There are 21 watchers for this library.

It had no major release in the last 6 months.

There are 1 open issues and 9 have been closed. On average issues are closed in 165 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of flume-kafka is current.

Quality

flume-kafka has 0 bugs and 0 code smells.

Security

flume-kafka has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

flume-kafka code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

flume-kafka does not have a standard license declared.

Check the repository for any license declaration and review the terms closely.

Without a license, all rights are reserved, and you cannot use the library in your applications.

Reuse

flume-kafka releases are not available. You will need to build from source code and install.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

flume-kafka saves you 93 person hours of effort in developing the same functionality from scratch.

It has 239 lines of code, 15 functions and 5 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed flume-kafka and discovered the below as its top functions. This is intended to give you an instant insight into flume-kafka implemented functionality, and help decide if they suit your requirements.

Set Kafka topic
Gets the Kafka config properties
Creates a producer
Returns the value of a Kafka config parameter
Configure the consumer
Creates a consumer
Process the incoming messages
Opens the event
Stop the producer
Stops the consumer

Get all kandi verified functions for this library.

flume-kafka Key Features

No Key Features are available at this moment for flume-kafka.

flume-kafka Examples and Code Snippets

No Code Snippets are available at this moment for flume-kafka.

Community Discussions

Trending Discussions on flume-kafka

Unable to configure KafkaChannel or KafkaSource in Flume for Kerberos enabled cluster-LoginException

Apache Flume-Kafka-Sink producer duplicated messages

QUESTION

Unable to configure KafkaChannel or KafkaSource in Flume for Kerberos enabled cluster-LoginException

Asked 2019-Jun-11 at 11:29

I try to setup KafkaChannel (or KafkaSource) in Flume. And I constantly receive following Exception

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. Make sure -Djava.security.auth.login.config property passed to JVM and the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using FQDN of the Kafka broker you are trying to connect to. not available to garner authentication information from the user

My jaas.conf is following:

...

ANSWER

Answered 2019-Jun-11 at 11:29

Thanks to this post (original) I've noticed that KafkaClient config specified in Flume 1.6 documentation provided by Cloudera was missing some options. Then I took a look at Official Apache Flume 1.7 documentation and noticed that I miss the following properties:

Source https://stackoverflow.com/questions/45083760

QUESTION

Apache Flume-Kafka-Sink producer duplicated messages

Asked 2018-Jan-30 at 09:13

We've meet strange problem with flume-kafka-sink, kafka broker failed multiple times and producing duplicate messages(every 50 record are same), but the settings about producer.sinks.r.request.required.acks = 1, quota to kafka documentation "This option provides the lowest latency but the weakest durability guarantees (some data will be lost when a server fails)", It can't be produce duplicate data? Is that means the problem caused by flume or flume-kafka-sink?

...

ANSWER

Answered 2018-Jan-30 at 09:13

Flume-Kafka-Sink produce messages batch by batch and will retry after some fail write. During some broker fail, some partition leaders can't reach. When a batch write happen, some parition will success, but some failed, when Flume-Kafka-Sink retry, the success part will be duplicated.

Source https://stackoverflow.com/questions/47096269

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install flume-kafka

You can download it from GitHub.
You can use flume-kafka like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the flume-kafka component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: