kafka-connect-elasticsearch | Kafka connector for copying data | Pub Sub library

First add more background. The way I deployed Kafka is using Strimzi:

In distributed mode, would I still want to submit just a single elasticsearch.properties through a single API call?

It'd be a JSON file, but yes.

what dictates the number of workers?

Up to you. JVM usage is one factor that you can monitor and scale on

Not really any documentation that I am aware of

Got this working and learned a few things in the process:

• Secret resources reside in a namespace. Secrets can only be referenced by Pods in that same namespace. (ref). Therefore, I switched to using a shared namespace for elasticsearch + kafka
• The secret can be used in a straightforward way as documented at https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets. This is not a Helm-specific but rather core Kubernetes feature

In my case this looked like:

The relevant line of code is here: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/src/main/java/io/confluent/connect/elasticsearch/DataConverter.java#L170

This basically means that the source document is sent as is to the index - no modifications.

Your best option is probably to add an SMT that reads the source document and removes any fields with null values.

Docker volumes are ephemeral, so this is expected behavior.

You need to mount host volumes for at least the Kafka and Zookeeper containers

e.g.

to install SMT it just the same as installing other connector,

Copy your custom SMT JAR file (and any non-Kafka JAR files required by the transformation) into a directory that is under one of the directories listed in the plugin.path property in the Connect worker configuration –

In your case copy to /usr/share/confluent-hub-components

The Connect container starts Connect Distributed Server already. You should use HTTP and JSON properties to configure the Elastic connector rather than exec into the container shell and issue connect-standalone commands which default to using a broker running in the container itself.

Similarly, the Elastic quickstart file expects Elasticsearch running within the Connect container, by default

curl -s localhost:8083/connector-plugins gives the definitive response from the worker what plugins are installed.

Per the output in your question, the Elasticsearch sink connector is now installed in your connector. I don't know why the Confluent CLI would not show this.

Yes you can do this. You can use a single Kafka cluster and single Kafka Connect worker. One connector can write to one Elasticsearch instance, and so if you have multiple destination Elasticsearch you need multiple connectors configured.

The usual way to run Kafka Connect is in "distributed" mode (even on a single instance), and then you submit one—or more—connector configurations via the REST API.

You don't need a Java client to use Kafka Connect - it's configuration only. The configuration, per connector, says where to get the data from (which Kafka topic(s)) and where to write it (which Elasticsearch instance).

To learn more about Kafka Connect see this talk, this short video, and this specific tutorial on Kafka Connect and Elasticsearch