cppkafka | Modern C++ Apache Kafka client library | Pub Sub library

I'm trying to setup a Consumer and it is taking 5 seconds before initialization finishes and messages start getting received. I am using CppKafka (librdkafka wrapper), but I suspect it's a matter of me misconfiguring things rather than a library problem.

I was previously using kafka inside of docker and the startup delay was 10 seconds. I am now using this quickstart on the Kafka website and the startup delay is now at 5 seconds.

I have a ConsumeOne function that looks like this.

I'm trying to use Kafka in a project that uses CMake to be built.

I have librdkafka-dev (version 0.8.6-1.1) installed.

I also tried downloading the project from Git and building it manually, which seem to be successful.

I noticed a README.md file in the librdkafka/packaging/cmake and followed its instructions, successfully creating a CMake build of librdkafka in the directory.

Running the 'cmake' command in my project results in the following error:

I am new to cmake and want to use the following external project in my program. (https://github.com/mfontanini/cppkafka) The idea is to download from git and build using cmake, which I have managed already. At the end of it, I am left with a .so file which I then have to link back to the main project. Any ideas on how I can achieve this ? Also, would like some suggestion on how to manage dependencies in a project with cmake that would possibly be carried forward to production.

I'm using the following C++ code to produce messages to Kafka:

I've developed a couple of C++ apps that produce and consume Kafka messages (using cppkafka) embedding Protobuf3 messages. Both work fine. The producer's relevant code is:

I am trying to send a video stream through a Kafka topic and doing some processing between each messages consumption. This processing takes approximately 20 milliseconds, so after 100 frames, I have a consequent latency. I can detect this latency, however, I need to be able to force my consumer to read from the latest sent frame from the moment the latency is detected. I am aware that this would lead to many frames skipped.

I am using cppkafka which is based on librdkafka.

I can't find a lead that will put me in the right path since most of the answers are either different from my problem or use concepts existing only in other languages APIs.

Any thought on the matter ?

I am using CppKafka to programming Kafka consumer. I want when my consumer starts, it will only poll new arrival messages (i.e message arrive after the consumer-start-time) instead of messages at consumer offset.

As mentioned in the cppkafka official GitHub in usage section.

If you want to use cppkafka, you'll need to link your application with:

• cppkafka
• rdkafka

I tried to compile example program mentioned in the README.md of the GitHub with

I have a kafka consumer that does a poll every 10 seconds. I am using wireshark to monitor my network's acitvity.

I noticed that even when I am not doing any fetch requests, there is still traffic between the broker and my consumer. And I also noticed that it is the same packets (almost the same, only a slight change to the payload) that are sent and received periodically.

Is this some kind of keep alive packets? how to reduce them?

Here is a screenshot of those packets:

PS: I am using cppkafka as lib and kafka broker 0.8.2.2

EDIT: The client's code