kafka | Mirror of Apache Kafka | Pub Sub library

I am new to kafka and zookepper, and I am trying to create a topic, but I am getting this error -

My buckets are:

• MyDataBucket: application saves its data on this bucket.
• MyEventingBucket: A couchbase eventing function extracts the 'currentState' field from MyDataBucket and saves it in this bucket.

Also, I have a kafka couchbase connector that pushs data from MyEventingBucket to kafka topic.

When we had a single data center, there wasn't any problem. Now, we have three data centers. We replicate our data with XDCR between data centers and we work as active-active. So, write requests can be from any data center.

When data is replicated on other data centers, the eventing service works on all data centers, and the same data is pushed three-time (because we have three data centers) on Kafka with Kafka connector.

How can we avoid pushing duplicate data o Kafka?

Ps: Of course, we can run an eventing service or Kafka connector in only one data center. So, we can publish data on Kafka just once. But this is not a good solution. Because we will be affected when a problem occurs in this data center. This was the main reason of using multi data center.

I'm running Kafka schema registry version 5.5.2, and trying to register a schema that contains a reference to another schema. I managed to do this when the referenced schema was in the same package with the referencing schema, with this curl command:

I was trying to build a new image for a small dotnet core 3.1 console application. I got an error:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to copy: httpReadSeeker: failed open: failed to do request: Get https://westeurope.data.mcr.microsoft.com/42012bb2682a4d76ba7fa17a9d9a9162-qb2vm9uiex//docker/registry/v2/blobs/sha256/87/87413803399bebbe093cfb4ef6c89d426c13a62811d7501d462f2f0e018321bb/data?P1=1627480321&P2=1&P3=1&P4=uDGSoX8YSljKnDQVR6fqniuqK8fjkRvyngwKxM7ljlM%3D&se=2021-07-28T13%3A52%3A01Z&sig=wJVu%2BBQo2sldEPr5ea6KHdflARqlzPZ9Ap7uBKcEYYw%3D&sp=r&spr=https&sr=b&sv=2016-05-31&regid=42012bb2682a4d76ba7fa17a9d9a9162: x509: certificate has expired or is not yet valid

I have checked an old dotnet program which my dockerfile was working perfectly. I got the same error. Then, I jumped to Docker Hub and checked the MS Images to see that all MS images have been updated for an hour. And then they have been updated once again, 10 Minutes ago xD. However, I still cannot pull the base images of mcr.microsoft.com/dotnet/runtime:3.1 and mcr.microsoft.com/dotnet/sdk:3.1. My whole Dockerfile is:

We have a bunch of microservices based on Spring Boot 2.5.4 also including spring-kafka:2.7.6 and spring-boot-actuator:2.5.4. All the services use Tomcat as servlet container and graceful shutdown enabled. These microservices are containerized using docker.
Due to a misconfiguration, yesterday we faced a problem on one of these containers because it took a port already bound from another one.
Log states:

I tried to run Kafka on CMD in Windows and it's very unstable , constantly giving errors. Then I came across this post, which suggests installing Ubuntu and run Kafka from there.

I have installed Ubuntu successfully. Given that I have already defined JAVA_HOME=C:\Program Files\Java\jdk1.8.0_231 as one of the environmental variables and CMD recognizes this variable but Ubuntu does not, I am wondering how to make Ubuntu recognize this because at the moment, when i typed java -version, Ubuntu returns command not found.

Update: Please note that I have to have Ubuntu's JAVA_HOME pointing to the evironmental variable JAVA_HOME defined in my Window system. Because my Java program in eclipse would need to talk to Kafka using the same JVM.

I have added the two lines below in my /etc/profile file. echo $JAVA_HOME returns the correct path. However, java -version returns a different version of Java installed on Ubuntu, not the one defined in the /etc/profile

I have the following code

We've been moving our applications from CircleCI to GitHub Actions in our company and we got stuck with a strange situation.

There has been no change to the project's code, but our kafka integration tests started to fail in GH Actions machines. Everything works fine in CircleCI and locally (MacOS and Fedora linux machines).

Both CircleCI and GH Actions machines are running Ubuntu (tested versions were 18.04 and 20.04). MacOS was not tested in GH Actions as it doesn't have Docker in it.

Here are the docker-compose and workflow files used by the build and integration tests:

• docker-compose.yml

I want to read json messages from Kafka and put them into another structure of SpecificRecordBase class (avro). The part of the json has dynamic structure for example