kafka-python | Python client for Apache Kafka | Pub Sub library

I had to set up partition. This snippet work for me.

Get topic partitions, assigned to your consumer:

That method doesn't accept timestamps; only partitions that you want to find timestamps for.

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.TopicPartition.TopicPartition

Perhaps you mean the timeout parameter?

You need the get_watermark_offsets() function of the Consumer. You call it with a list of TopicPartition and it returns a tuple (int, int) (low, high) for each partition.

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.Consumer.get_watermark_offsets

Something like this:

If you literally ran the Python sample code, then the key is not Avro, so a failure on the key.converter would be expected, as shown

Error converting message key

Jsonschema support was not added to the Confluent Schema Registry until version 6.0, and this is why the error reports issues about a schemaType field, because any lower versions of the Registry response/request payload do not know about that field.

Upgrading to at least that version, or using the latest version of the image will solve that error

If you just want to produce JSON, then you don't need the Registry. More details at https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/ .
You can use the regular producer.py example and provide JSON objects as strings on the CLI

The confluent protobuf serializer doesn't produce content that can be directly deserialized by other deserializers. The format is described in confluent's documentation: it starts with a magic byte (that is always zero), followed by a four byte schema ID. The protobuf payload follows, starting with byte 5.

The getProducedType method should return appropriate TypeInformation, in this case TypeInformation.of(User.class). Without this you may run into problems at runtime.

Deserializers used with KafkaSource don't need to implement isEndOfStream, but it won't hurt anything.

Read the error message, where it is clearly stated you are missing a header file:

fatal error: rocksdb/slice.h: No such file or directory 705 | #include "rocksdb/slice.h" | ^~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1

Accordingly, you'll need to build and install RocksDB. This is separate from the installation of faust[rocksdb] with pip. That simply installs python-rocksdb, the Python interface to the underlying libraries.

There is even a (third-party) RocksDB docker image based on Python 3.7 Slim.

You could use that directly or take some tricks from the Dockerfile for that image.

Kafka Connect is a Java Framework, not Python.

Kafka Connect runs a REST API which you can use urllib3 or requests to interact with it, not kafka-python

https://kafka.apache.org/documentation/#connect

Once you create a connector, you are welcome to use kafka-python to produce data, which the JDBC sink would consume, for example, or you can use pandas for example to write to a database, which the JDBC source (or Debezium) would consume