kafka-streams | equivalent to kafka-streams octopus for nodejs | Stream Processing library

I am facing a weird issue with when aggregating records into a Ktable. I have a following scenario in my system.

1. There are two kafka streams application running on different nodes (having the same application id but having different application server config).

2. Both of these streams listen to the same topic pattern where the records are partitioned by a key (string value).

3. Whenever both the application is running , some partition are consumed by app-1 and some are consumed by app-2 which is normal. They then build their own respective local state store.

4. I have a grapql query system which lets you query the key and get its value if its in local table or in another remote instance.

5. The problem is that when I query for a key's metadata it is giving me the wrong hostInfo (even if the key is processed by instance one it shows it has hostInfo of instance two) But when if I query the key's value in instance-1's local state store I can see that the key is indeed present. (It just that the metadata for the key's is wrong)

6. And this behaviour is random for key in both instance (some keys point the correct metadata while some don't)

7. I have logged for a state listener which tells me if a rebalancing is happening or not. But while the records are streaming or when I am querying , I have make sure that no rebalancing is happening.

8. The issue I face is something similar to this. Method of metadataForKey in Kafka Streams gives wrong information for multiple instances of application connected to the same group

9. Also when I query for all keys in the local state store I can see the key is present.

Anyone have idea of what could be causing this issue? Please

I currently have a table in KSQL which created by

The following error would appear every 5 seconds when we have Kafka running on Windows 10.

Failed to write offset checkpoint file to C:/tmp/kafka-streams/user/global/.checkpoint for global stores: {}. This may occur if OS cleaned the state.dir in case when it is located in the (default) ${java.io.tmpdir}/kafka-streams directory. Changing the location of state.dir may resolve the problem.

java.nio.file.AccessDeniedException: C:/tmp/kafka-streams/user/global

Here is our Gradle. At the time of writing, this will import the 3.0 version by default.

I'm facing an issue with Spring Kafka which is that it cannot access state store from process event I added that particular store into topology/streams.

method 1:

Initial Question: I have a question how I can bind my GlobalStateStore to a processor. My Application has a GlobalStateStore with an own processor ("GlobalConfigProcessor") to keep the Store up to date. Also, I have another Processor ("MyClassProcessor") which is called in my Consumer Function. Now I try to access the store from MyClassProcessor, but I get an exception saying : Invalid topology: StateStore config_statestore is not added yet.

Update on current situation: I setup a test repository to give a better overview over my situation. This can be found here: https://github.com/fx42/store-example

As you can see in the repo, I have two Consumers which both consume different topics. The Config-Topic provides an event which I want to write to a GlobalStateStore. Here are the StateStoreUpdateConsumer.java and the StateStoreProcessor.java involved. With the MyClassEventConsumer.java I process another Input-Topic and want to read values from the GlobalStateStore. As provided in this doc I can't initialize GlobalStateStores just as StateStoreBean but instead I have to add this actively with the StreamsBuilderFactoryBeanCustomizer Bean. This Code is currently commented out in the StreamConfig.java. Without this code I get the Exception

It's my first Kafka program.

From a kafka_2.13-3.1.0 instance, I created a Kafka topic poids_garmin_brut and filled it with this csv:

In order to try the Kafka stream I did this :

I am using Kafka 2.6 with spring cloud stream kafka-streams binder. I want to access record headers, partition no etc in my Kafka streams application. I read about using Processor API, using ProcessorContext etc. But everytime ProcessorContext object is coming null.

Below is the code

I have a fresh Spring Boot 2.6.3 Java 11 application with a Spring Kafka Dependency (generated with start.spring.io).

By default Kafka 3.0.0 is is used. I want to change the Kafka version to 3.1.0 and added