zookeeper | The learning process of distributed system service ZooKeeper | Messaging library

Known bug on the Apache Kafka side. Nothing to do from Spring perspective. See more info here: https://github.com/spring-projects/spring-kafka/discussions/2027. And here: https://issues.apache.org/jira/browse/KAFKA-13391

You need to wait until Apache Kafka 3.0.1 or don't use embedded Kafka and just rely on the Testcontainers, for example, or fully external Apache Kafka broker.

Read the official Kafka documentation for the version you downloaded, and not some other blog/article that you might have copied the command from

zookeeper is almost never used for CLI commands in current versions

If you run bin\kafka-topics on its own with --help or no options, then it'll print the help messaging that shows all available arguments.

I'd like to start with a disclaimer that I'm personally not familiar with AWS MSK offering in great detail so this answer is largely based on my understanding of the open source distribution of Apache Kafka.

First - The Kafka ACLs are actually stored in Zookeeper by default so if you're not using Zookeeper, it might be worth adding this if you're not using it.

Reference - Kafka Definitive Guide - 2nd edition - Chapter 11 - Securing Kafka - Page 294

Second - If you're using SASL for authentication through any of the supported mechanisms such as GSSAPI (Kerberos), then you'll need to create a principal as you would normally create one and use one of the following options:

1. Add the required permissions for topic creation/deletion etc. using the kafka-acls command (Command Reference)

   bin/kafka-acls.sh --add --cluster --operation Create --authorizer-properties zookeeper.connect=localhost:2181 --allow-principal User:admin

   Note - admin is the assumed principal name

2. Or add admin user to the super users list in server.properties file by adding the following line so it has unrestricted access on all resources

   super.users=User:Admin

   Any more users can be added in the same line delimited by ;.

To add the strictness, you'll need to set allow.everyone.if.no.acl.found to false so any access to any resources is only granted by explicitly adding these permissions.

Third - As you've asked specifically about your root user, I'm assuming you're referring to the linux root here. You could just restrict the linux level permissions using chmod command for the kafka-acls.sh script but that is quite a crude way of achieving what you need. I'm also not entirely sure if this is doable in MSK or not.

Update: The answer is that the KafkaSource behaves differently than FlinkKafkaConsumer in the case where the number of Kafka partitions is smaller than the parallelism of Flink's kafka source operator. See https://stackoverflow.com/a/70101290/2000823 for details.

Original answer:

The problem is almost certainly something related to the timestamps and watermarks.

To verify that timestamps and watermarks are the problem, you could do a quick experiment where you replace the 3-hour-long event time sliding windows with short processing time tumbling windows.

In general it is preferred (but not required) to have the KafkaSource do the watermarking. Using forMonotonousTimestamps in a watermark generator applied after the source, as you are doing now, is a risky move. This will only work correctly if the timestamps in all of the partitions being consumed by each parallel instance of the source are processed in order. If more than one Kafka partition is assigned to any of the KafkaSource tasks, this isn't going to happen. On the other hand, if you supply the forMonotonousTimestamps watermarking strategy in the fromSource call (rather than noWatermarks), then all that will be required is that the timestamps be in order on a per-partition basis, which I imagine is the case.

As troubling as that is, it's probably not enough to explain why the windows don't produce any results. Another possible root cause is that the test data set doesn't include any events with timestamps after the first window, so that window never closes.

Do you have a sink? If not, that would explain things.

You can use the Flink dashboard to help debug this. Look to see if the watermarks are advancing in the window tasks. Turn on checkpointing, and then look to see how much state the window task has -- it should have some non-zero amount of state.

We identified some test sequence dependency between the Kafka tests.

We updated our Gradle version to 7.3-rc-3 which has a more deterministic approach to test scanning. This update "solved" our problem while we prepare to fix the tests' dependencies.

Finally, the solution for the issue is to create appropriate dns records for each cluster ec2 instance with the necessary names inside Amazon Route53 service.

With your shown samples, attempts please try following awk code. Since I don't have zookeeper command with me, I had written this code and tested it as per your shown output only.

You've forwarded the wrong port

9093 on the host needs to map to the localhost:9093 advertised port

Otherwise, you're connecting to 9093, which returns kafka:9092, as explained in the blog. Container hostnames cannot be resolved by the host, by default

AS written in the documentation:

Additionally, containers that belong to the same task can communicate over the localhost interface.

So my suggestion is to use localhost instead of the service names. For example, you want to do it for Kafka but also for every other service, such as the email service.