kafkalog | nullkafkalog bridges a syslog feed | Pub Sub library

by Multifarious Java Version: 1 License: BSD-2-Clause

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | kafkalog Summary

kafkalog is a Java library typically used in Messaging, Pub Sub, Kafka applications. kafkalog has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub, Maven.

kafkalog

Support

Quality

Security

License

Reuse

Support

kafkalog has a low active ecosystem.

It has 4 star(s) with 0 fork(s). There are 5 watchers for this library.

It had no major release in the last 12 months.

kafkalog has no issues reported. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of kafkalog is 1

Quality

kafkalog has no bugs reported.

Security

kafkalog has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

kafkalog is licensed under the BSD-2-Clause License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kafkalog releases are not available. You will need to build from source code and install.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed kafkalog and discovered the below as its top functions. This is intended to give you an instant insight into kafkalog implemented functionality, and help decide if they suit your requirements.

The main entry point
Creates a producer with the given configuration
Entry point for processing
Gets the list of available brokers
Returns the maximum length of the line
Gets the max queue length
Get the port number
Gets the topic
Receives a message
Get channel pipeline
Start the syslog listener
Check if the producer task is healthy
Called when an exception occurs
Shutdown the channel
Start the producer task
Stops the producer task

Get all kandi verified functions for this library.

kafkalog Key Features

No Key Features are available at this moment for kafkalog.

kafkalog Examples and Code Snippets

No Code Snippets are available at this moment for kafkalog.

Community Discussions

Trending Discussions on kafkalog

I want to record STORM log by log4j and send log to kafka. the result is kafka STORM can record log by itself log4j but not send log to kafka

Apache Kafka - recurring reblancing with more consumers

QUESTION

I want to record STORM log by log4j and send log to kafka. the result is kafka STORM can record log by itself log4j but not send log to kafka

Asked 2019-Sep-16 at 20:47

I'm using ubuntu14.6,kafka_2.11-2.20,zookeeper-3.4.14,storm-1.2.3.when i use LocalCluster pattern can send log to kafka, i can get the log by kafka consumer. but i tend to submit topology by cluster pattern , it can't get log by kafka consumer. this is my log4j configuration

...

ANSWER

Answered 2019-Sep-16 at 20:47

I don't think this question really has anything to do with Storm. I don't think the issue is with log4j either. You probably need to check that your distributed worker machines can actually connect to Kafka.

Also keep in mind that you will want to put your log4j config in storm/log4j2/worker.xml.

Source https://stackoverflow.com/questions/57951736

QUESTION

Apache Kafka - recurring reblancing with more consumers

Asked 2019-Jan-22 at 14:29

Context

We are using Kafka to process large messages, very occasionally up to 10MB but mostly in the 500 KB range. The processing of a message can take up to around 30 seconds, but sometimes a minute.

The Issue

Processing the data with a lower amount of consumers (up to around 50) causes no recurring re-balancing by the broker and the processing works fine. Any re-balancing at this scale is also rather fast, mostly under a minute as per the broker logs.

Once the consumers are scaled to 100 or 200, the consumers constantly re-balance, with intervals up to around 5 minutes. This results into 5 minutes working/consuming followed by 5 minutes re-balancing and then the same again. The consumer services don't fail, just re-balance for no real apparent reason. This leads to reduced throughput when scaling consumers up.

When scaled to 2oo consumers, the processing performs at an average rate of 2 messages per minute per consumer. The processing speed for a single consumer when it is not re-balancing is around 6 messages per minute.

I don't suspect the network of the data centers to be an issue as we have some consumers performing a different kind of processing on the messages and they have no issues passing 100's to 1000's of messages per minute.

Did someone else experience this pattern and find a simple solution, e.g. changing a particular configuration parameter?

Additional Information

The Kafka brokers are version 2.0, and there are 10 of them across different data centers. Replication is set to 3. Partitions for this topic are 500. An excerpt of the specific broker configuration to suit the case of processing large messages better:

compression.type=lz4
message.max.bytes=10000000 # 10 MB
replica.fetch.max.bytes=10000000 # 10 MB
group.max.session.timeout.ms=1320000 # 22 min
offset.retention.minutes=10080 # 7 days

On the consumer side we use the java client with a re-balance listener that clears up any buffered messages from the revoked partitions. This buffer is 10 messages large. The consumer clients run client API version 2.1, the update of the java client from 2.0 to 2.1 seems to significantly reduce broker logs of the following kind on these larger consumer numbers (we got these for almost every client and every re-balance before):

...

ANSWER

Answered 2019-Jan-22 at 14:29

After some further engineering and fine-tuning, we managed to get the issue under control.

First, it seems that some services still processed above the limit and this caused them to fail very rarely. The following leaving caused a re-balance, followed by the joining after around 6-7 minutes, which caused a re-balance too. We reduced this further by optimizing our services in terms of throughput.

A second factor was the underlying docker network that we use to scale the services up. By default the heartbeat interval is very short (5 seconds), so that any hard work and network load on the consumer node may remove it from the docker swarm for a very brief interval. This interruption is answered by docker with moving these services to other nodes (re-balancing), followed by re-balancing when the node comes back online. As the services have long startup times of 5-7 minutes, this leads to a few times of re-balancing on each of these events.

A third factor were errors in the consuming services that caused one of them to crash occasionally, say 1% per hour. This causes again two re-balances, one leaving, one joining.

Collectively, these issues combined led to the observed issue. The latest Kafka version also seems to output more information on why a service is leaving the consumer group. It would be nice if Kafka would continue serving data to consumers that are still stable, I may add a feature request around this. Nevertheless, we have it running stable now with decent performance.

Source https://stackoverflow.com/questions/53462206

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kafkalog

You can download it from GitHub, Maven.
You can use kafkalog like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the kafkalog component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .