kafka-service | Spring Boot Application for your Kafka testing needs | Pub Sub library

We faced a similar issue in one of our Kafka setups; we ended up creating two k8s services, one using ClusterIP for internal communication and second service with same labels using NodePort for external communication.

internal access

I have been reading this blog post "Kafka Listeners - Explained" and I was able to configure 3 Kafka brokers with the following configuration.

You used a service with type = LoadBalancer which is used for cloud provider (you can see the service waiting for an external ip address .. pending state ... which will never happen). In your case you should try with NodePort.

My initial guess would be that you might be trying to receive a request that is too large. The maximum size is the default size for socket.request.max.bytes, which is 100MB. So if you have a message which is bigger than 100MB try to increase the value of this variable under server.properties.

Then make sure that your nodes have proper seletector: node for deployment broker has to have role:slave1 seletector. Otherwise just delete lines with nodeSelector from broker deployment file.

Then add lines to spec of your deployments configuration file:

I had a similar issue when I was trying to access kafka broker (cp-helm-chart) running on minikube from outside.

Here how I resolved it. Before you install using helm install from local repository.

1. Edit inside this file https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/values.yaml
2. Search for nodeport: and change its enabled field to true.
   nodeport:
   enabled: true
3. Uncomment these two lines by removing #:
   "advertised.listeners": |-
   EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))
4. Replace ${HOST_IP} by your minikube ip (enter minikube ip in cmd to retrieve your k8s host ip e.g : 196.169.99.100)
5. Replace ${KAFKA_BROKER_ID} by the broker id (if only one broker is running than its will be by default just 0)
6. Finally it would look something like this:
   "advertised.listeners": |-
   EXTERNAL://196.169.99.100:31090

Now you can access the kafka broker running within k8s cluster from outside by pointing the bootstrap.servers to 196.169.99.100:31090

The YAML indentation on the serverless.yml is incorrect. This means the event properties aren't registered by the framework during deployment.

Change the serverless.yml file to the following format and it should work.

The error is from the deserialiser trying to connect to the Schema Registry

props.put("schema.registry.url", "http://0.0.0.0:8081")

should read

props.put("schema.registry.url", "http://:8081")

Thank you @SoheilPourbafrani and @cricket_007 for your help! I have found the workaround for the question I asked above.

Once I run the below command in the window PowerShell, Kafka started properly and able to communicate with it from Node Application and Kafka Tool as well.