karafka | Ruby and Rails efficient Kafka processing framework | Stream Processing library

by karafka Ruby Version: v2.1.4 License: Non-SPDX

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | karafka Summary

karafka is a Ruby library typically used in Data Processing, Stream Processing, Ruby On Rails, Kafka applications. karafka has no bugs, it has no vulnerabilities and it has medium support. However karafka has a Non-SPDX License. You can download it from GitHub.

Framework used to simplify Apache Kafka based Ruby applications development. Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming and processing this data, without having to focus on things that are not your business domain. Karafka not only handles incoming messages but also provides tools for building complex data-flow applications that receive and send messages.

Support

Quality

Security

License

Reuse

Support

karafka has a medium active ecosystem.

It has 1731 star(s) with 157 fork(s). There are 32 watchers for this library.

There were 5 major release(s) in the last 12 months.

There are 24 open issues and 641 have been closed. On average issues are closed in 50 days. There are 3 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of karafka is v2.1.4

Quality

karafka has 0 bugs and 0 code smells.

Security

karafka has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

karafka code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

karafka has a Non-SPDX License.

Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

Reuse

karafka releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed karafka and discovered the below as its top functions. This is intended to give you an instant insight into karafka implemented functionality, and help decide if they suit your requirements.

Initialize the listener
Validates that the response is received .
Validates the message and raises an error message
attempts to a topic .
Initialize the consumer .
Reloads the given loader .
Reloads reloads .
Get the consumer
Dispatch the response .
Issues a signal to notice .

Get all kandi verified functions for this library.

karafka Key Features

No Key Features are available at this moment for karafka.

karafka Examples and Code Snippets

No Code Snippets are available at this moment for karafka.

Community Discussions

Trending Discussions on karafka

Long delays between processing of two consecutive kafka batches (using ruby/karafka consumer)

QUESTION

Long delays between processing of two consecutive kafka batches (using ruby/karafka consumer)

Asked 2021-Jun-24 at 13:49

I am using karafka to read from a topic, and call an external service. Each call to external service takes roughly 300ms. And with 3 consumers (3 pods in the k8s) running in the consumer group, I expect to achieve 10 events per second. I see these loglines , which also confirm the 300ms expectation for processing each individual event. However, the overall throughput doesn't add up. Each karafka processes seems stuck for a long time between processing two batches of events.

Following instrumentation around the consume method, implies that the consumer code itself is not taking time.

https://github.com/karafka/karafka/blob/master/lib/karafka/backends/inline.rb#L12

...

ANSWER

Answered 2021-Jun-24 at 08:27

There are a couple of problems you may be facing. It is a bit of a guessing from my side without more details but let's give it a shot.

From the Kafka perspective

Are you sure you're evenly distributing data across partitions? Maybe it is eating up things from one partition?

What you wrote here:

INFO Inline processing of topic production.events with 8 messages took 2571 ms

This indicates that there was a batch of 8 processed altogether by a single consumer. This could indicate that the data is not distributed evenly.

From the performance perspective

There are two performance properties that can affect your understanding of how Karafka operates: throughput and latency.

Throughput is the number of messages that can be processed in a given time
Latency is the time it takes a message from the moment it was produced to it been processed.

As far as I understand, all messages are being produced. You could try playing with the Karafka settings, in particular this one: https://github.com/karafka/karafka/blob/83a9a5ba417317495556c3ebb4b53f1308c80fe0/lib/karafka/setup/config.rb#L114

From the logger perspective

Logger that is being used flushes data from time to time, so you won't see it immediately but after a bit of time. You can validate this by looking at the log time.

Source https://stackoverflow.com/questions/68108380

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install karafka

If you're completely new to the subject, you can start with our "Kafka on Rails" articles series, that will get you up and running with the terminology and basic ideas behind using Kafka:.
Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages
Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Ruby and Kafka

Support

Karafka has Wiki pages for almost everything and a pretty decent FAQ. It covers the whole installation, setup, and deployment along with other useful details on how to run Karafka. If you have any questions about using Karafka, feel free to join our Gitter chat channel.

Find more information at: