confluent-schema-registry | is a library that makes it easier to interact | Pub Sub library

(end goal) before trying out whether i could eventually read avro data, usng spark stream, out of the Confluent Platform like some described here: Integrating Spark Structured Streaming with the Confluent Schema Registry

I'd to verify whether I could use below command to read them:

I am trying to build an infrastructure in which I need to forward messages from one kafka topic to elasticsearch and postgresql. My infrastructure looks like in the picture below, and it all runs on the same host. Logstash is making some anonymization and some mutates, and sends the document back to kafka as json. Kafka should then forward the message to PostgreSQL and Elasticsearch

Everything works fine, accept the connection to postgresql, with which i'm having some trouble.

My config files looks like follows:

sink-quickstart-sqlite.properties

I just finished this tutorial to use Kafka and Schema Registry :http://cloudurable.com/blog/kafka-avro-schema-registry/index.html I also played with Conlfuent Platform : https://docs.confluent.io/current/installation/installing_cp.html

Everything works fine, until I rebooted my Virtual Machine (VMBOX) : All schemas/subjects have been deleted (or disappeared) after I rebooted.

I read that Schema Registry to not store itself the data but use Kafka to do that. Of course, as I work for the moment only on my laptop, Kafka was also shutdown during the machine reboot.

Is it normal behavior, do we have to expect to RE-store all schemas all the time we reboot??? (-> maybe last version so!)

Do anybody have good best practices about that?

How persistence of schemas can be managed to avoid this problem ?

Environment : Ubuntu 16... , Kafka 2.11.1.0.0, Confluent Platform 4.0

Thanks a lot

nota: I already read this topics which discuss about keeping schema's ID, but has I don't recover any schemas, it's not a problem of Ids : Confluent Schema Registry Persistence

I'm following the instructions linked in this wiki doc to install the confluent platform on my EC2 instance running amazon linux (version 2016.09). I did everything it says including: