KafkaClient | A C Kafka client for librdkafka 0.8 and some articles | Pub Sub library

I found the error, I should've been encoding the partition key and not the json id:

Print the partition and offset of the messages. You should see they are, in fact, unique events you're processing.

If those are the same, the "10min to 4hr" process is very likely causing a consumer group rebalance (Kafka requires you to invoke a record poll every few milliseconds, by default), and you're experiencing at-least-once processing semantics, and therefore need to handle duplicates on your own.

I see you're using some database client in your code, and so the recommendation would be to use Kafka Connect framework, rather than writing your own Consumer

You will get this error message when Jaas cannot access the kerberos keytab.

Can you check for user permission issues? Login as user that will run the code and do a kinit ? What error message do you get? (Resolve the permission issue I'm suggesting you have.)

You seem to rule out a path issue, and seem to have the correct '\\'.

please help me understand why i cannot mock getEnvironment function properly ?

In the test file, replace getEnvironment.mockResolvedValue('testBrokers') with getEnvironment.mockReturnValue('testBrokers') as the former is used to mock async functions.

Refer official documentation for more details

Make sure you mock kafkaBrokers look up for "testBrokers" as well for expected results

brokers: kafkaBrokers[getEnvironment()]

is there any difference in mocking a default imported function and mocking a named imported function ?

Ideally there are no difference in mocking them. Refer this interesting jest feature for more insights.

While org.apache.kafka.common.network.SslChannelBuilder provides reconfigure method, it appears to be used only by the Kafka broker code.

In case of clients, it looks like you'd need to restart them, as the ChannelBuilder instance is configured only once, at startup.

Reference (Kafka 2.8):

• ChannelBuilders
• SSLChannelBuilder

Finally the issue was fixed. I am updating the fix here so that it can be beneficial for someone:

kafka-rest.properties file should have below text:

do not use "kafka-node". replace with "kafkajs"

You can try filtering WARN logs using the below snippet in log4j.xml

Structr 3.6.4 is compatible with Neo4j 3.x.

Our current 4.0-SNAPSHOT builds support Neo4j 4.x.

As we are pretty close to a release of 4.0 and it contains many new features and fixes, I would recommend you use a SNAPSHOT version if you are not running on production yet.