artemis | minimalist apollo replacement | GraphQL library

ActiveMQ Artemis has no default connection limit. I just wrote a quick test based on this which uses the Paho 1.2.5 MQTT client. It spun up 500 concurrent connections using both normal TCP and WebSockets. The test finished in less than 20 seconds with no errors. I'm just running this test on my laptop.

I noticed that your journal-buffer-timeout is 700000 which seems quite high which means you have a very low write speed of 1.43 writes per millisecond (i.e. a slow disk). The journal-buffer-timeout that is calculated, for example, on my laptop is 4000 which translates into a write-speed of 250 which is significantly faster than yours. My laptop is nothing special, but it does have an SSD. That said, SSDs are pretty common. If this low write-speed is indicative of the overall performance of your VM it may simply be too weak to handle the load you want. To be clear, this value isn't related directly to MQTT connections. It's just something I noticed while reviewing your configuration that may be indirect evidence of your issue.

The journal-buffer-timeout value is calculated and configured automatically when the instance is created. You can re-calculate this value later and configure it manually using the bin/artemis perf-journal command.

Ultimately, your issue looks environmental to me. I recommend you inspect your VM and network. TCP dumps may be useful to see perhaps how/why the connection is being reset. Thread dumps from the server during the time of the trouble would also be worth inspecting.

There are a couple of important points to understand before getting started:

• When using the Artemis Prometheus Metrics Plugin neither the broker nor the applications "send" metrics to Prometheus. Prometheus itself must retrieve or "scrape" metrics from the broker. This is why the plugin comes with a servlet. The servlet exposes an HTTP endpoint that Prometheus can use to scrape metrics.
• The Artemis Prometheus Metrics Plugin is part of the broker infrastructure. It is not to be deployed as part of an application. The plugin's jar and war files are deployed on the broker and configured in broker.xml and bootstrap.xml respectively.

The Artemis Prometheus Metrics Plugin provides integration with Prometheus using two modules:

• artemis-prometheus-metrics-plugin: This provides the actual implementation of org.apache.activemq.artemis.core.server.metrics.ActiveMQMetricsPlugin and packages it with the Micrometer and Prometheus dependencies in an "uber" jar.

• artemis-prometheus-metrics-plugin-servlet: This provides a war file containing a simple servlet which can be deployed to the broker's embedded web server which then Prometheus can use to scrape metrics.

Once you clone the Artemis Prometheus Metrics Plugin repository simply run mvn install to build these two modules. The output will be in their respective target directories.

After building the modules follow these steps to deploy and configure the Artemis Prometheus Metrics Plugin. If you have some kind of dev-ops group which manages and configures your broker then they would follow these steps.

1. Copy artemis-prometheus-metrics-plugin/target/artemis-prometheus-metrics-plugin-.jar to /lib.

2. Add this to your /etc/broker.xml:

The 2.6.3.redhat-00015 version corresponds to AMQ 7.2.3 which is quite old at this point. The current AMQ release is 7.8.1. I strongly recommend you upgrade as it's likely you're hitting a bug that's already been fixed.

You may be able to work around the issue by increasing the minimum large message size (e.g using minLargeMessageSize on core client URLs or amqpMinLargeMessageSize on your AMQP acceptor). For what it's worth, the stack-trace indicates that the core JMS client (i.e. not AMQP) is in use when the exception is thrown.

Lastly, it's worth noting that the default minimum large message size is 100 KB not 2 GB as explained in the documentation.

Given that you want clients to be able to connect and:

• specify a selector for the data they need
• only receive new incoming messages

That means that you want to use a JMS topic.

However, your @JmsListener definitions are using a JMS queue instead, because that's what they use by default. Take a look at this answer for details on how to make them use a JMS topic.

Since your @JmsListener definitions are using a JMS queue the broker is auto-creating and using anycast resources automatically. This is why you see the same behavior no matter what configuration you change on the broker.

Ultimately you don't need to define any address or queue in broker.xml. As long as the client is using the right kind of JMS resources all the broker-side resources will be created automatically. Also, your @JmsListener definition should just use the name of the address and not the FQQN.

By default the broker will auto-create addresses and queues as required when a message is sent or a consumer is created by the core JMS client. These resources will also be auto-deleted by default when they're no longer needed (i.e. when a queue has no consumers and messages or when an address no longer has any queues bound to it). This is controlled by these settings in broker.xml which are discussed in the documentation:

• auto-create-queues
• auto-delete-queues
• auto-create-addresses
• auto-delete-addresses

To be clear, auto-deletion should not cause any message loss by default as queues should only be deleted when they have 0 consumers and 0 messages. However, you can always set auto-deletion to false to be 100% safe.

Queues representing durable JMS topic subscriptions won't be deleted as they are meant to stay and gather messages while the consumer is offline. In other words, a durable topic subscription will remain if the client using the subscription is shutdown without first explicitly removing the subscription. That's the whole point of durable subscriptions - they are durable. Any client can use a durable topic subscription if it connects with the same client ID and uses the same subscription name. However, unless the durable subscription is a "shared" durable subscription then only one client at a time can be connected to it. Shared durable topic subscriptions were added in JMS 2.0.

it says you are trying to assign or add a JMS message body INTO an "event class". Events usually have a named overridden method for code to be placed inside to run that code if the event is triggered.

It may be this here , the compiler error message seems to indicate "resp" , a message body and "response" ApplicationsEvent.

The RoutingContext object, which is used internally by the broker, is reusable. This is done for performance reasons to prevent having to re-create the RoutingContext for every routing operation no matter what. As one might guess, routing messages is a very common operation in the broker so it pays to optimize it as much as possible. Reusing the RoutingContext means fewer objects are created and thrown away which means less garbage needs to be cleaned up which means fewer pauses and better overall performance by the broker.

The fact that the previousAddress is different here from the address where the current message is going to be routed is not a problem. It just means that the context won't be re-used for this routing operation and therefore will be cleared. As the name suggests, the beforeMessageRoute method is invoked before any routing logic is performed (e.g. clearing the RoutingContext). If you inspect the RoutingContext using afterMessageRoute then you should see that it was cleared and populated with the proper details.

Message "sending" and message "routing" (both of which have plugin hooks) are related but distinct operations. A message is "sent" in response to a client operation. Sends always result in a route. However, not all routes are the results of sends. A message can be routed due to internal broker operations which do not involve a send (e.g. moving messages around a cluster, expiring a message, cancelling an undeliverable message to a dead-letter address, using a divert, etc.).

I would caution you against inspecting internal broker state (which can be subtle and nuanced) and assuming a problem exists when everything else indicates that the broker is functioning normally. In this case you said that you were "facing routing issues" and that "some message are routed from topic.private.abc.task.V1 to topic.abc.rawmessage.V1" when, in fact, there was no routing issue and messages were not actually being routed from topic.private.abc.task.V1 to topic.abc.rawmessage.V1. From what I can see everything is in fact functioning normally.

I've taken your simplified configured with just 2 nodes using a non-wildcard queue with redistribution-delay of 0, and I reproduced the behavior you're seeing on my local machine (i.e. without Kubernetes). I believe I see why the behavior is such, but in order to understand the current behavior you first must understand how redistribution works in the first place.

In a cluster every time a consumer is created the node on which the consumer is created notifies every other node in the cluster about the consumer. If other nodes in the cluster have messages in their corresponding queue but don't have any consumers then those other nodes redistribute their messages to the node with the consumer (assuming the message-load-balancing is ON_DEMAND and the redistribution-delay is >= 0).

In your case however, the node with the messages is actually down when the consumer is created on the other node so it never actually receives the notification about the consumer. Therefore, once that node restarts it doesn't know about the other consumer and does not redistribute its messages.

I see you've opened ARTEMIS-3321 to enhance the broker to deal with this situation. However, that will take time to develop and release (assuming the change is approved). My recommendation to you in the mean-time would be to configure your client reconnection which is discussed in the documentation, e.g.:

In Result object with ID 385687 you have a property backdrop_path being null. Adjust your Result object and make the property nullable:

String? backdropPath;