logstash-input-kafka | Kafka input for Logstash | Pub Sub library

You set logstash to connect to Zookeeper, not Kafka.

Initialize connection to node localhost:2181

Make sure the bootstrap server is localhost:9092, in your case

The problem was in the Dockerfile itself which was missing && in between RUN commands which was therefore removing the logstash-plugin from the folder instead of executing it to install a plugin.

Correct Dockerfile:

You need to use localhost:9092 for configuration parameter bootstrap_servers - you're pointing Logstash to Zookeeper, but you need to point to Kafka itself.

Another error is that topics requires an array, so it should be ["test"] instead of "test"

First you must install the logstash input plugin by running

bin/logstash-plugin install logstash-input-jdbc

And also If it does not help, try

Java::com.mysql.jdbc.Driver

instead of the

com.mysql.jdbc.Driver

for the jdbc_driver_class. I had a same kind of problem with oracle when I did not use the

Java::

before the oracle.jdbc.OracleDriver as the jdbc_driver_class.

You most probably have a version conflict that is causing this issue. Check out the compatibility matrix in the Logstash Kafka input plugins documentation.

The link you mentioned for installing Kafka has you install version 0.8.2.1 which will not work with Kafka 0.10 clients. Kafka has version checking and backwards compatibility, but only if the broker is newer than the client, which is not the case here. I'd recommend installing a current version of Kafka, there have been immense improvements since version 0.8 that you'd be missing out on if you tried downgrading Logstash instead.

Check out the Confluent Platform Quickstart for an easy way to get started.

The default serializer was changed in logstash 5 from byte array deserializer to string deserializer.

https://www.elastic.co/guide/en/logstash/2.4/plugins-inputs-kafka.html#plugins-inputs-kafka-consumer_threads

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-consumer_threads

Please add below config in kafka input:
key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer" value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"