librdkafka | The Apache Kafka C/C++ library | Pub Sub library

by edenhill C Version: v1.9.2 License: Non-SPDX

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | librdkafka Summary

librdkafka is a C library typically used in Messaging, Pub Sub, Kafka applications. librdkafka has no bugs, it has no vulnerabilities and it has medium support. However librdkafka has a Non-SPDX License. You can download it from GitHub.

Copyright (c) 2012-2020, [Magnus Edenhill] librdkafka is a C library implementation of the [Apache Kafka] protocol, providing Producer, Consumer and Admin clients. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer. librdkafka is licensed under the 2-clause BSD license. KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by librdkafka. librdkafka has no affiliation with and is not endorsed by The Apache Software Foundation.

Support

Quality

Security

License

Reuse

Support

librdkafka has a medium active ecosystem.

It has 6380 star(s) with 2896 fork(s). There are 314 watchers for this library.

It had no major release in the last 12 months.

There are 159 open issues and 2667 have been closed. On average issues are closed in 40 days. There are 54 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of librdkafka is v1.9.2

Quality

librdkafka has no bugs reported.

Security

librdkafka has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

librdkafka has a Non-SPDX License.

Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

Reuse

librdkafka releases are available to install and integrate.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of librdkafka

Get all kandi verified functions for this library.

librdkafka Key Features

No Key Features are available at this moment for librdkafka.

librdkafka Examples and Code Snippets

No Code Snippets are available at this moment for librdkafka.

Community Discussions

Trending Discussions on librdkafka

How to consume messages from a specific partition of a topic in kafka

Kerberros GSSAPI doesn't work within kafkacat alpine container

Logging kubernetes container log to azure event hub using fluent-bit - error while loading shared libraries: librdkafka.so

How To Implement A Wait Retry Transient Fault Handling Policy?

ClickHouse Kafka Table Engine and exactly once support

Does librdkafka and confluent-kafka-go supports JKS based SSL Configuration?

Problem with ADVERTISED_LISTENER on macos

Why can't find RdKafka?

What does librdkafka.redist's logs mean

How to access librdkafka.redist logs from C# application which uses it as a dependency?

QUESTION

How to consume messages from a specific partition of a topic in kafka

Asked 2021-Jun-10 at 12:06

I want a kafka consumer to consume messages from a particular/specified partition of a topic

It works using kafka-console-consumer.sh with switch --partition partition_number

I am using kafka_complex_consumer_example.c code from librdkafka

Upon static code analysis, I feel it can serve my purpose but I am not able to find out the exact command line parameters to be passed to main(int argc & char **argv) function for the code to run and start consuming on a topic's particular partition.

Have a look at the code here - rdkafka_complex_consumer_example.c

Full github code of librdkafka here FYR

If this code doesn't serve the purpose, please specify some other code that can help

...

ANSWER

Answered 2021-Jun-10 at 12:06

If you look at the usage string, putting a colon after the topic name designates which partition to consume from

More specifically, topicpartition type holds information for a specific partition, and this line creates the list of those

https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_complex_consumer_example.c#L528

Source https://stackoverflow.com/questions/67920566

QUESTION

Kerberros GSSAPI doesn't work within kafkacat alpine container

Asked 2021-May-13 at 11:50

Previously I've reported it into kafkacat tracker but the issue has been closed as related to cyrus-sasl/krb5.

...

ANSWER

Answered 2021-May-13 at 11:50

Very strange issue, and honestly I can't say why, but adding into krb5.conf:

Source https://stackoverflow.com/questions/67509575

QUESTION

Logging kubernetes container log to azure event hub using fluent-bit - error while loading shared libraries: librdkafka.so

Asked 2021-Apr-13 at 15:24

$ kubectl create namespace logging

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/0.13-dev/output/kafka/fluent-bit-configmap.yaml

After this getting error

/fluent-bit/bin/fluent-bit: error while loading shared libraries: librdkafka.so: cannot open shared object file: No such file or directory

And fluent-bit pod is creating but in CrashLoopBackOff

Ref - https://github.com/fluent/fluent-bit-kubernetes-logging

Can any one suggest how to resolve this

...

ANSWER

Answered 2021-Apr-13 at 15:24

Two things:

Deploy all from master branch. You seem to be installing Kafka configmap from developer branch while the rest is from master.
You skipped to create a Kafka deamonset. Do below.

$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-ds.yaml

Source https://stackoverflow.com/questions/67067614

QUESTION

How To Implement A Wait Retry Transient Fault Handling Policy?

Asked 2021-Mar-25 at 11:53

I am fairly new to Kafka and Polly. I am seeking advice with respect to how to implement failure resiliency when using the Admin Client with Kakfa Confluent .NET client. I am using the Admin Client to create a topic if it does not already exist during startup of a Blazor Server Web App.

Initially, I am trying to use polly to implement a simple wait and retry policy, listed below. I am expecting this to retry a create topic operation for a configurable number of attempts. Between each retry attempt there is a short configurable wait delay. If all retry attempts have been exhausted then a fatal error is signalled and the application gracefully exits.

Wait and Retry Policy

...

ANSWER

Answered 2021-Mar-25 at 11:53

After filing an issue at the Confluent Kafka GitHub repository it looks as though the problem described in this question is due to a confirmed bug in the Confluent Kafka .NET library.

Workaround is suggested by the libraries author here.

Essentially until the bug is fixed, a new AdminClient instance has to be created for each retry attempt.

Source https://stackoverflow.com/questions/65509865

QUESTION

ClickHouse Kafka Table Engine and exactly once support

Asked 2021-Jan-29 at 00:18

I'm looking for some clarity as to if the Kafka Table engine supports exactly once semantics. I understand that clickhouse uses librdkafka, and that librdafka supports EOS as of v1.4. The latest versions of clickhouse are using librdkafka v1.5+. Is use of the library enough to confirm that EOS is supported by the kafka table engine, or does the table engine functionality require additional changes to support EOS? It isn't clear to me from the clickhouse documentation.

...

ANSWER

Answered 2021-Jan-29 at 00:18

No EO with Clickhouse until two-phase-commit is implemented in CH.

https://github.com/ClickHouse/ClickHouse/issues/18668#issuecomment-752946654

BTW - while EOS semantics will guarantee you that no duplicates will happen on the Kafka side (i.e. even if you produce the same messages few times it will be consumed once), but ClickHouse as a kafka client can guarantee only at-least-once. And in some corner cases (connection lost etc) you can get duplicates. We need to have something like transactions on clickhouse side to be able to avoid that.

Source https://stackoverflow.com/questions/65942650

QUESTION

Does librdkafka and confluent-kafka-go supports JKS based SSL Configuration?

Asked 2020-Dec-07 at 12:05

Can we configure librdkafka and confluet-kafka-go to use jks files

...

ANSWER

Answered 2020-Dec-07 at 12:05

No, JKS is a Java-specific format. See Converting a Java Keystore into PEM Format for how to extract certificates in PEM format from a JKS.

Source https://stackoverflow.com/questions/65118705

QUESTION

Problem with ADVERTISED_LISTENER on macos

Asked 2020-Nov-14 at 16:00

I start kafka with this docker-compose.yml on my Mac:

...

ANSWER

Answered 2020-Nov-12 at 14:55

I played around with your particular example, and couldn't get it to work.

For what it's worth, this Docker Compose is how I run Kafka on Docker locally, and it's accessible both from the host machine, and other containers.

You might find this blog useful if you want to continue with your existing approach and debug it further.

Source https://stackoverflow.com/questions/64805874

QUESTION

Why can't find RdKafka?

Asked 2020-Oct-14 at 19:26

I have a dockerfile:

...

ANSWER

Answered 2020-Oct-14 at 19:26

You need to install ext-rdkafka.

Source https://stackoverflow.com/questions/64123756

QUESTION

What does librdkafka.redist's logs mean

Asked 2020-Oct-12 at 10:32

This is an extract from a console log relating to Confluent.Kafka's librdkafka.redist dependency.

...

ANSWER

Answered 2020-Oct-12 at 10:32

Columns:

%3 - syslog severity level. Lower is more severe.
1602097315.970 - seconds since epoch.
rdkafka#consumer-2 - client instance name, which is a combination of the (possibly configured) client.id, the client type (consumer) and a running number that is increased (in the current process) for each new client instance.
[thrd:kfkqaapq0002d.ch.me.com:9092/bootstrap] - what thread the log is emitted from. Application threads will be named thrd:app, all other thread names are librdkafka internal threads.
kfkqaapq0002d.ch.me.com:9092/bootstrap - the broker the log message corresponds to.
Failed to resolve 'kfkqaapq0002d.ch.me.com:9092': No such host is known. (after 42ms in state CONNECT) - the log message itself.

Source https://stackoverflow.com/questions/64258309

QUESTION

How to access librdkafka.redist logs from C# application which uses it as a dependency?

Asked 2020-Oct-09 at 06:15

My application console log shows a lot of the following:

...

ANSWER

Answered 2020-Oct-09 at 06:15

You can pass your own log handler to the client builder, see https://docs.confluent.io/current/clients/confluent-kafka-dotnet/api/Confluent.Kafka.ProducerBuilder-2.html#Confluent_Kafka_ProducerBuilder_2_SetLogHandler_System_Action_Confluent_Kafka_IProducer__0__1__Confluent_Kafka_LogMessage__

E.g.:

Source https://stackoverflow.com/questions/64251115

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install librdkafka

You can download it from GitHub.

Support

Public API in [C header](src/rdkafka.h) and [C++ header](src-cpp/rdkafkacpp.h). Introduction and manual in [INTRODUCTION.md](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md). Configuration properties in [CONFIGURATION.md](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). Statistics metrics in [STATISTICS.md](https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md). NOTE: The master branch is actively developed, use latest [release](https://github.com/edenhill/librdkafka/releases) for production use.

Find more information at: