kafka-node | Node.js client for Apache Kafka | Stream Processing library

do not use "kafka-node". replace with "kafkajs"

OK, I figured out the issue. I thought the error was telling me that path was undefined. When it fact it was saying the variables passed into path.join() were undefined. And that was because I forgot to add my .env file to the root so it could grab those variables.

Since it was an enterprise project so they don't keep .env file in the source code, I asked them and put it root.

The import type syntax used in apollo-server-core isn't supported by the version of typescript that you're using (v3.6). This syntax became available from v3.8 onwards. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-8.html#type-only-imports-and-export

Update typescript and the error will disappear

As @OneCricketeer told above If you ship Kafka code to clients, then you need to update your advertised listeners on the brokers... But running Kafka clients in a browser isn't really possible

It's best to use websockets, so I have made a micro service (NodeJs) for this to communicate with the backend server and my client-side will communicate with that micro service(NodeJS) using socket-io

This approach while better seems a bit off since I will have to make a consumer every time a user clicks on a page, and I only care about it being sent 1 message.

I would come to the same conclusion. Cassandra is not designed for these kind of use cases. The database is eventually consistence. Your current approach maybe works at the moment, if you hack something together, but will definitely result in undefined behavior once you have a Cassandra cluster. Especially when you update the entry.

The id in the computation table is your partition key. This means once you have a cluster Cassandra distributes the data by the id. It looks like it only contains one row. This is a very inefficient way of modeling your Cassandra tables.

Your use case looks like one for a session storage or cache. Redis or LevelDB are well suited for these kind of use cases. Any other key value storage would do the job too.

Why don't you write your result into another topic and have another application which reads this topic and writes the result into a database. So that you don't need to keep any state. The result will be in the topic when it is done. It would look like this:

incoming data -> first kafka topic -> computational app -> second kafka topic -> another app writing it into the database <- another app reading regularly the data.

If it is there it is there and therefore not done yet.

You are "globally" defining the Leader to be on broker with id 0 whereas you want to have the partitions 1 and 2 located on other brokers. As you defined the replication to be one, this is contradicting itself. Remove the part about the Leader and it should automatically create the partitions leaders on the brokers you want.

Here you used createTopics() on kafka server and it will only work when auto.create.topics.enable, on the Kafka server, is set to true. Client simply sends a metadata request to the server which will auto create topics. When async is set to false, this method does not return until all topics are created, otherwise it returns immediately. So, here default one topic with one partition is creating. To create multiple partition or to make it customize you have to add following line in server.property file -

According the to documentation on a ProducerRecord you can specify the partition of a ProducerRecord. That way you can write the same message to multiple partitions of the same topic. The api for this look like this in Java:

You exposed port 5434 of your container, but PostgreSQL itself is still configured to listen on port 5432. That is why you don't reach the database.

After running initdb and before starting PostgreSQL, configure the cluster, for example with