kafka-streams-scala | Thin Scala wrapper around Kafka Streams Java API | Pub Sub library

I'm implementing mechanism which gives some metadata information by requesting it locally from store or requesting remote Kafka Streams instance.

I'm using Scala and kafka-streams-scala library of version 2.4.1

I'll try to give you small simple example of what I'm doing

1. I'm running the Kafka cluster which creates 1 test topic with 2 partitions.
2. As well I run 1 Kafka Streams instance as I mentioned above that implements mechanism of requesting local or remote metadata from store and it holds all the partitions information until no any other instances are connected to the same group.
3. I push some records into test topic

When upgrading from KafkaStreams library from 2.3 to 2.5, keeping the same Scala version, it breaks when running with the following error:

We are having these dependencies:

I am very new to using Microservices and having trouble running Kafka after I have started zookeeper.

Zookeeper starts fine but when I try to start my Kafka server it throws an error.

I have searched on google to try and solve my problem but its quite overwhelming, as I am not sure what all these different config files mean/do.

I have tried by enabling listeners=PLAINTEXT://:9092 in server settings but it doesn't work.

I have also tried to un and reinstalled Kafka and ZooKeeper but I still get the same error.

I made a jar file for a Kafka Streams application coded in Scala using both IntelliJ IDEA itself and then SBT. Following are the problems I have been facing and so far I haven't succeeded:

First, I made a jar file using IntelliJ IDEA. Upon running the Jar file, it gave the following error:

java.lang.ClassNotFoundException: Main ...

Following are the steps I took to make the Jar file in IntelliJ IDEA:

• Selected Project Structure from the file menu.
• Went into the 'Artifacts' tab.
• Clicked on the + button.
• Selected the main class and the META-INF directory location
• Added all the Scala libraries and dependencies (and made sure the META-INF dir was included)
• Applied the settings and built the artifact.
• Run the Jar using Scala command. Gave the aforementioned error.

After that I tried making the Jar file using SBT. I made an assembly.sbt file inside the project directory and then I reimported the project. After that, I executed the following commands:

• sbt clean
• sbt package
• sbt assembly

It made a Jar file. This time, the problem was that it wasn't picking some of the Kafka libraries randomly. Sometimes, it'll throw, for example, that KafkaStreams not found. Then I'll try to rebuild the jar and the new jar would then throw StreamsBuilder not found. I kept on trying and to sum up, following are some of the errors I remember:

NoClassDefFoundException ..../KafkaStreams
NoClassDefFoundException ..../StreamsBuilder etc.

Here '...' means the full package name/path.

build.sbt: