kafka-best-practices | repo discusses some techniques of consuming kafka | Pub Sub library

by sceneryback Go Version: Current License: MIT

X-Ray Key Features Code Snippets Community Discussions(3)Vulnerabilities Install Support

kandi X-RAY | kafka-best-practices Summary

kafka-best-practices is a Go library typically used in Messaging, Pub Sub, Unity, Kafka applications. kafka-best-practices has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

This repo discusses some techniques of consuming kafka (Sync, Batch, MultiAsync and MultiBatch) and try to demonstrate some best practices which I think would be generally useful to consume data efficiently. consume messages one by one. consume messages batch by batch. the "Fan In / Fan Out" pattern. the "Fan In / Fan Out" pattern batch by batch.

Support

Quality

Security

License

Reuse

Support

kafka-best-practices has a low active ecosystem.

It has 112 star(s) with 31 fork(s). There are 4 watchers for this library.

It had no major release in the last 6 months.

There are 1 open issues and 2 have been closed. On average issues are closed in 35 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of kafka-best-practices is current.

Quality

kafka-best-practices has no bugs reported.

Security

kafka-best-practices has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

kafka-best-practices is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kafka-best-practices releases are not available. You will need to build from source code and install.

Top functions reviewed by kandi - BETA

kandi has reviewed kafka-best-practices and discovered the below as its top functions. This is intended to give you an instant insight into kafka-best-practices implemented functionality, and help decide if they suit your requirements.

Basic example of Kafka
NewConsumerGroup creates a new consumer group
StartMultiBatchConsumer creates a new consumer group
StartMultiAsyncConsumer creates a new async consumer group
StartBatchConsumer starts a new consumer group
NewBatchConsumerGroupHandler creates a new consumer group handler
NewMultiBatchConsumerGroupHandler creates a new multiBatchConsumerGroupHandler .
StartSyncConsumer creates a new sync consumer group
NewProducer returns a new producer
NewMultiAsyncConsumerGroupHandler returns a new consumer group handler .

Get all kandi verified functions for this library.

kafka-best-practices Key Features

No Key Features are available at this moment for kafka-best-practices.

kafka-best-practices Examples and Code Snippets

No Code Snippets are available at this moment for kafka-best-practices.

Community Discussions

Trending Discussions on kafka-best-practices

Kafka using Docker for production clusters

Kafka Best Practices + how to set recommended setting for JVM

How to update an existing Storm topology with new bolts in code?

QUESTION

Kafka using Docker for production clusters

Asked 2019-Oct-29 at 12:05

We need to build a Kafka production cluster with 3-5 nodes in cluster ,

We have the following options:

Kafka in Docker containers (Kafka cluster include zookeeper and schema registry on each node)
Kafka cluster not using docker (Kafka cluster include zookeeper and schema registry on each node)

Since we are talking on production cluster we need good performance as we have high read/write to disks (disk size is 10T), good IO performance, etc.

So does Kafka using Docker meet the requirements for productions clusters?

more info - https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment/

...

ANSWER

Answered 2019-Oct-29 at 03:12

It can be done, sure. I have no personal experience with it, but if you don't otherwise have experience managing other stateful containers, I'd suggest avoiding it.

As far as "getting started" with Kafka in containers, Kubernetes is the most documented way, and Strimzi (free, optional commercial support by Lightbend) or Confluent Operator (commercial support by Confluent) can make this easy when using Kubernetes or Openshift. Or DC/OS offers a Kafka service over Mesos/Marathon. If you don't already have any of these services, then I think it's apparent that you should favor not using containers.

Bare metal or virtualized deployments would be much easier to maintain than hand-deployed containerized ones, from what I have experienced. Particularly for logging, metric gathering, and statically assigned Kafka listener mappings over the network. Confluent provides Ansible scripts for doing deployments to such environments

That isn't to say there's companies that have been successful at it, or at least tried. IBM, RedHat, and Shopify immediately pop up in my searches, for example

Here's a few talk about things to consider when Kafka is in containers https://www.confluent.io/kafka-summit-london18/kafka-in-containers-in-docker-in-kubernetes-in-the-cloud

https://kafka-summit.org/sessions/running-kafka-kubernetes-practical-guide/

Source https://stackoverflow.com/questions/58600130

QUESTION

Kafka Best Practices + how to set recommended setting for JVM

Asked 2019-Mar-13 at 11:31

A recommended setting for JVM looks like following

...

ANSWER

Answered 2018-May-25 at 09:05

As you mentioned, -Xmx8G -Xms8G should be set using KAFKA_HEAP_OPTS.

For the other configurations you listed, you should probably use KAFKA_JVM_PERFORMANCE_OPTS.

I'm not aware of a place where all the supported environment variables are clearly described. The best is to check the kafka-run-class.sh tool, as it is called by all the tools, including kafka-server-start.sh.

For example:

KAFKA_HEAP_OPTS is used in https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L232-L234
KAFKA_JVM_PERFORMANCE_OPTS is used in https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L236-L239

Source https://stackoverflow.com/questions/50524643

QUESTION

How to update an existing Storm topology with new bolts in code?

Asked 2018-Aug-20 at 16:30

I'm writing a dockerized Java Spring application that uses Apache Storm v1.1.2, Kafka v0.11.0.1, Zookeeper 3.4.6, Eureka, and Cloud-Config all in Docker containers orchestrated by Docker-Compose.

The tuples I'm receiving with a KafkaSpout have a "value" Field that is a protobuf object. I use a custom deserializer to get my object out of it for processing.

I have a basic application working where I have a bolt that prints incoming messages and routes them to other certain bolts based on the value of a field in the protobuf object. I also have the LocalCluster, Config, and TopologyBuilder working as Spring Beans.

Currently I set all bolts in a PostContruct but I need to be able to dynamically add bolts that filter incoming messages based on other fields of the protobuf object and perform basic aggregation functions (max/min/windowed average).

I'd like to do this with a REST Controller but how could I stop and start the topology without losing data? I also would prefer not to restart the topology by listening to the Kafka topic from the beginning as this system will receive an extremely high load.

This article looked promising but I definitely want the entire process to be automated so I won't be going into Zookeeper https://community.hortonworks.com/articles/550/unofficial-storm-and-kafka-best-practices-guide.html

How can I edit an existing topology in code to add new bolts dynamically?

...

ANSWER

Answered 2018-Aug-20 at 16:30

You can't. Storm topologies are static once submitted. If you need to vary processing based on a field in the tuple, your best option is to submit all the bolts you will need up front. You can then vary the path the tuple takes through the topology by using one or more bolts that examine the tuple, and emit to specific streams based on the tuple content.

e.g. make a SplitterBolt

Source https://stackoverflow.com/questions/51874584

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kafka-best-practices

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: