kafkacat | Generic command line non-JVM Apache Kafka producer | Pub Sub library

To use the Kafka Connect JDBC Sink, your data needs an explicit schema. The converter that you specify in your connector configuration determines where the schema is held. This can either be embedded within the JSON message (org.apache.kafka.connect.json.JsonConverter with schemas.enabled=true) or held in the Schema Registry (one of io.confluent.connect.avro.AvroConverter, io.confluent.connect.protobuf.ProtobufConverter, or io.confluent.connect.json.JsonSchemaConverter).

To learn more about this see https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained

To write an Avro message to Kafka you should serialise it as Avro and store the schema in the Schema Registry. There is a Go client library to use with examples

without getting Confluent getting involved.

It's not entirely clear what you mean by this. The Kafka Connect JDBC Sink is written by Confluent. The best way to manage schemas is with the Schema Registry. If you don't want to use the Schema Registry then you can embed the schema in your JSON message but it's a suboptimal way of doing things.

the producer is treating each line as a new message

That's correct.

If you have a binary file, I suggest writing code for this, as kafkacat assumes UTF8 encoded strings as input

I played around with your particular example, and couldn't get it to work.

For what it's worth, this Docker Compose is how I run Kafka on Docker locally, and it's accessible both from the host machine, and other containers.

You might find this blog useful if you want to continue with your existing approach and debug it further.

Broker configuration seems to be fine since you get back the correct metadata.

I think the problem is in your code. kafkaTemplate.send() is an asynchronous operation and most likely your process ends before the producer manages to actually send the message. Try adding a .get() to that send method to force it in being synchronous.

kafkacat does not yet support the Topic Admin API (which allows you to alter and view cluster configs). Suggest you use kafka-configs.sh from the Apache Kafka distribution in the meantine.

Found out the answer:

Use kubefwd CLI

You're using the wrong port to connect to the broker. By connecting to 9092 you connect to the listener that advertises localhost:9092 to the client for subsequent connections. That's why it works when you use kafkacat from your local machine (because 9092 is exposed to your local machine)

If you use broker:29092 then the broker will give the client the correct address for the connection (i.e. broker:29092).

To understand more about advertised listeners see this blog

Not clear why you need a JAR file. This should work just as well

A KSQL table differs from a KSQL Stream, in that it gives you the latest value for a given key. So if you are expecting to see the same number of messages in your table as your source stream, you should have the same number of unique keys.

If you're seeing fewer messages then it suggests that ROOT is not unique.

Depending on the problem that you're modelling, you should either :

• (a) be using a Stream not a Table, or
• (b) change the key that you are using

Ref:

• https://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables
• http://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1/