broker-plugin | client apps and WP-based servers | OAuth library

There is definitely a problem with your configuration. The error message even says so:

What actually happened!

On the Connect cluster this appears when all nodes on the cluster have entered into what Confluent calls a 'STOP-THE-WORLD' event of heavy re-balancing.

Which essentially means no matter how many connector workers / tasks were running on the cluster before, they all stop processing whatever they were before and jump into re-balancing mode fighting for a leader.

Why it happened!

One of your Connect worker properties file is set to this -> connect.protocol=compatible

OR

Some other major change in the connect worker properties or worker restart without pausing the running tasks first

Solution

ActiveMQ Artemis is failing because the bind attribute of the web element is using the HTTPS protocol:

I set up a simple pair of brokers using replication and the "Broker Diagram" displays both the master and the slave as expected: Ensure you have the "Show Backup Brokers" checkbox selected.

Regarding the SSL issue, see the upgrade instructions for 2.18.0 in the documentation:

...core clients will now expect the CN or Subject Alternative Name values of the broker's SSL certificate to match the hostname in the client's URL.

...

To deal with this you can do one of the following:

• Update your SSL certificates to use a hostname which matches the hostname in the client's URL. This is the recommended option with regard to security.

• Update any connector using sslEnabled=true to also use verifyHost=false. Using this option means that you won't get the extra security of hostname verification, but no certificates will need to change. This essentially restores the previous default behavior.

ActiveMQ Artemis has no default connection limit. I just wrote a quick test based on this which uses the Paho 1.2.5 MQTT client. It spun up 500 concurrent connections using both normal TCP and WebSockets. The test finished in less than 20 seconds with no errors. I'm just running this test on my laptop.

I noticed that your journal-buffer-timeout is 700000 which seems quite high which means you have a very low write speed of 1.43 writes per millisecond (i.e. a slow disk). The journal-buffer-timeout that is calculated, for example, on my laptop is 4000 which translates into a write-speed of 250 which is significantly faster than yours. My laptop is nothing special, but it does have an SSD. That said, SSDs are pretty common. If this low write-speed is indicative of the overall performance of your VM it may simply be too weak to handle the load you want. To be clear, this value isn't related directly to MQTT connections. It's just something I noticed while reviewing your configuration that may be indirect evidence of your issue.

The journal-buffer-timeout value is calculated and configured automatically when the instance is created. You can re-calculate this value later and configure it manually using the bin/artemis perf-journal command.

Ultimately, your issue looks environmental to me. I recommend you inspect your VM and network. TCP dumps may be useful to see perhaps how/why the connection is being reset. Thread dumps from the server during the time of the trouble would also be worth inspecting.

First, it's important to note that there's no feature to make a client reconnect to the broker from which it disconnected after the client crashes/restarts. Generally speaking the client shouldn't really care about what broker it connects to; that's one of the main goals of horizontal scalability.

It's also worth noting that if the number of messages on the brokers and the number of connected clients is low enough that this condition arises frequently that almost certainly means you have too many brokers in your cluster.

That said, I believe the reason that your client isn't getting the messages it expects is because you're using the default redistribution-delay (i.e. -1) which means messages will not be redistributed to other nodes in the cluster. If you want to enable redistribution (which is seems like you do) then you should set it to >= 0, e.g.:

It's really up to the subscriber to delete its subscription when it is done with it. In other words, the MQTT client should unsubscribe the existing subscriber before creating a new one. The broker has no way to know whether or not the subscriber plans to reconnect later to fetch the messages from its subscription.

If you really want the broker to delete the subscription queue you can use the following address-settings:

• : This must be true.
• : This is the delay (in milliseconds) from the time the last consumer disconnects until the broker deletes the subscription queue.
• : This must be -1 in order to ignore any messages in the subscription queue.

To be clear, I would recommend against this configuration as the broker may inadvertently delete legitimate messages. For example, if a subscriber mistakenly disconnects (e.g. due to a network failure, hardware failure, JVM crash, etc.) for longer than the configured then the broker will delete its subscription queue and all the subscriber's messages will effectively be lost.

Using 0.0.0.0 for a connector is not valid. In fact, there should be a WARN level message in the log about it. Your cluster will not form properly with such a configuration. You need to use a hostname or IP address that other hosts can use to reach the broker that broadcasts it. That's the whole point of the broadcast-group - to tell other brokers who may be listening how they can connect to it.