cp-all-in-one | docker-compose.yml files for cp-all-in-one , cp-all-in-one-community, cp-all-in-one-cloud, Apache Ka

I'm building a Kafka Streams application using Protobuf for message schemas. For now the application itself is just piping from one topic to another. I'm running Kafka locally using the Confluent platform all-in-one docker-compose file.

One of my schemas (foo.proto) uses a Struct field, so prior to starting my app I have registered both foo.proto and struct.proto on the schema registry.

When I start my app the protobuf serializer runs a method called resolveDependencies, leading it to re-register subtruct.proto. The (local) schema registry returns a 409 with message:

I'm using docker to run kafka and other services from https://github.com/confluentinc/cp-all-in-one with confluent nuget packages for kafka, avro and schemaRegistry in my test project.

If it goes to sending json messages I have no problem till now, but I'm struggling with sending avro serialized messages.

I saw https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/examples/AvroSpecific example and I tried to do it the same way but eventually I get an exception like below:

Local: Value serialization error
at Confluent.Kafka.Producer2.d__52.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Kafka_producer.KafkaService.d__10.MoveNext() in C:\Users\lu95eb\source\repos\Kafka_playground\Kafka producer\KafkaService.cs:line 126

with inner exception

Object reference not set to an instance of an object.
at Confluent.SchemaRegistry.Serdes.SpecificSerializerImpl1..ctor(ISchemaRegistryClient schemaRegistryClient, Boolean autoRegisterSchema, Int32 initialBufferSize) at Confluent.SchemaRegistry.Serdes.AvroSerializer1.d__6.MoveNext() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Confluent.Kafka.Producer`2.d__52.MoveNext()

Here's my SpecificRecord class

I have set up NiFi (1.11.4) & Kafka(2.5) via docker (docker-compose file below, actual NiFi flow definition https://github.com/geoHeil/streaming-reference). When trying to follow up on basic getting started tutorials (such as https://towardsdatascience.com/big-data-managing-the-flow-of-data-with-apache-nifi-and-apache-kafka-af674cd8f926) which combine processors such as:

• generate flowfile (CSV)
• update attribute
• PublishKafka2.0

I run into issues of timeoutException:

We are having some troubles with Spring Cloud and Kafka, at sometimes our microservice throws an UnkownProducerIdException, this is caused if the parameter transactional.id.expiration.ms is expired in the broker side.

My question, could it be possible to catch that exception and retry the failed message? If yes, what could be the best option to handle it?

I have took a look at:
- https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
- Kafka UNKNOWN_PRODUCER_ID exception

We are using Spring Cloud Hoxton.RELEASE version and Spring Kafka version 2.2.4.RELEASE

We are using AWS Kafka solution so we can't set a new value on that property I mentioned before.

Here is some trace of the exception:

I deployed Kafka from here https://github.com/confluentinc/examples/blob/5.3.1-post/cp-all-in-one/docker-compose.yml. But now I can't understand how to create persistent query. All I see is the init page:

In documentation https://docs.confluent.io/current/control-center/ksql.html it is not said how to get access.

What do I miss?

Thanks!

I deployed Kafka from here. Also I added to docker-compose.yml Postgres container like this:

I gave ksqldb a try and made an docker-compose.yml like this:

I'm trying to run the Confluent Platform all in one example using Docker Compose. The example of using it with a single node is here:

http://docs.confluent.io/3.1.1/cp-docker-images/docs/quickstart.html#getting-started-with-docker-compose

The git repository with all the Docker images also has a load of other examples, including one which is supposed to provide the Control panel etc, as detailed here: http://docs.confluent.io/3.1.2/cp-docker-images/docs/intro.html#choosing-the-right-images.

Running the simple example works fine. When I try to run the cp-all-in-one example (link to GitHub), I get the following error on running sudo docker-compose start (sudo docker-compose create runs without error):

I use All-In-One Confluent Platform https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html

I performed the steps described in the documentation above and was able to run Confluent Platform on Windows 10 machine via docker-compose up -d command on the following docker-compose.yml https://github.com/confluentinc/cp-docker-images/tree/master/examples/cp-all-in-one.

Everything is working fine except the error message I see in the console of my application: