confluent | xCAT confluent - replacement of conserver and eventually xcatd | Build Tool library

With org.springframework.kafka:spring-kafka up to version 2.7.9, my Spring-Boot application (consuming/producing Avro from/to Kafka) starts fine, having these environment variables set:

I'm trying to use Kafka Streams to perform KTable-KTable foreign key joins on CDC data. The data I will be reading is in Avro format, however it is serialized in a manner that wouldn't be compatible with other industry serializer/deserializers (ex. Confluent schema registry) because the schema identifiers are stored in the headers.

When I setup my KTables' Serdes, my Kafka Streams app runs initially, but ultimately fails because it internally invokes the Serializer method with byte[] serialize(String topic, T data); and not a method with headers (ie. byte[] serialize(String topic, Headers headers, T data) in the wrapping serializer ValueAndTimestampSerializer. The Serdes I'm working with cannot handle this and throw an exception.

First question is, does anyone know a way to implore Kafka Streams to call the method with the right method signature internally?

I'm exploring approaches to get around this, including writing new Serdes that re-serialize with the schema identifiers in the message itself. This may involve recopying the data to a new topic or using interceptors.

However, I understand ValueTransformer has access to headers in the ProcessorContext and I'm wondering if there might there be a faster way using transformValues(). The idea is to first read the value as a byte[] and then deserialize the value to my Avro class in the transformer (see example below). When I do this however, I'm getting an exception.

I tried to run Kafka on CMD in Windows and it's very unstable , constantly giving errors. Then I came across this post, which suggests installing Ubuntu and run Kafka from there.

I have installed Ubuntu successfully. Given that I have already defined JAVA_HOME=C:\Program Files\Java\jdk1.8.0_231 as one of the environmental variables and CMD recognizes this variable but Ubuntu does not, I am wondering how to make Ubuntu recognize this because at the moment, when i typed java -version, Ubuntu returns command not found.

Update: Please note that I have to have Ubuntu's JAVA_HOME pointing to the evironmental variable JAVA_HOME defined in my Window system. Because my Java program in eclipse would need to talk to Kafka using the same JVM.

I have added the two lines below in my /etc/profile file. echo $JAVA_HOME returns the correct path. However, java -version returns a different version of Java installed on Ubuntu, not the one defined in the /etc/profile

I need to exclude slf4j dependency from io.confluent:kafka-schema-registry:5.3.0 . I have tried using

I am using Avro serialiser to push messages to kafka topic. I generated the Java class out of the below avro schema

I'm trying to have a simple foreign key join in Kafka Streams similar to many articles (like this for one: https://www.confluent.io/blog/data-enrichment-with-kafka-streams-foreign-key-joins/).

When I try to join the user id (primary key of user table) with the foreign key user_id in the account_balance table to produce an AccountRecord object, I get the following error: [-StreamThread-1] ignJoinSubscriptionSendProcessorSupplier : Skipping record due to null foreign key.

The goal is ultimately to deliver the AccountRecord's to a topic each time any field in either table update. The problem is that when I simply print the user table and the account table separately, the foreign keys and all fields are totally populated. I can't see what's wrong or why this error occurs. Here is a snippet of my code:

I am trying to consume messages from a Kafka cluster external to my organization, which requires authentication.

I am receiving messages, so presumably things are partly correct, but I'm getting this error message in the logs:

08:54:50.840 [kafka-admin-client-thread | adminclient-1] ERROR i.m.m.health.indicator.HealthResult - Health indicator [kafka] reported exception: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

And a resulting status of DOWN in the health checks.

Here is the kafka section from application.yaml:

I'm building a provided Google Dataflow template here. So I'm running the command:

In continuation to my previous question C# Confluent.Kafka SetValueDeserializer object deserialization, I have tried creating my custom deserializer to deserialize protobuf message but getting this error: