rebalance | A portfolio rebalancing tool | Portfolio library

I've taken your simplified configured with just 2 nodes using a non-wildcard queue with redistribution-delay of 0, and I reproduced the behavior you're seeing on my local machine (i.e. without Kubernetes). I believe I see why the behavior is such, but in order to understand the current behavior you first must understand how redistribution works in the first place.

In a cluster every time a consumer is created the node on which the consumer is created notifies every other node in the cluster about the consumer. If other nodes in the cluster have messages in their corresponding queue but don't have any consumers then those other nodes redistribute their messages to the node with the consumer (assuming the message-load-balancing is ON_DEMAND and the redistribution-delay is >= 0).

In your case however, the node with the messages is actually down when the consumer is created on the other node so it never actually receives the notification about the consumer. Therefore, once that node restarts it doesn't know about the other consumer and does not redistribute its messages.

I see you've opened ARTEMIS-3321 to enhance the broker to deal with this situation. However, that will take time to develop and release (assuming the change is approved). My recommendation to you in the mean-time would be to configure your client reconnection which is discussed in the documentation, e.g.:

It took me some time to understand your problem and identify what was correct (the AVL part) and what was untested (in fact main code).

The problem is that newItem is declared outside of the while loop, so it keeps its values between iterations, specificaly the index member. So for a trivial fix, just declare it inside the loop:

It behaves as expected for me:

In order to implement a timeout, you can use Promise.race like this:

You can use StartableAllocationStrategy for your custom MyShardAllocationStrategy. Also, you need to change the type of shardAllocationStrategy variable to LeastShardAllocationStrategy.

Full code for reference:

Your observation is correct; the error handling is bound to the thread.

You could use a DeadLetterPublishingRecoverer directly in your code to make publishing the DLQ a litter easier (instead of an output channel). That way, you'll get the enhanced headers with exception information etc.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#dead-letters

EDIT

Here is an example; I am pausing the binding to prevent any new deliveries while the "job" is being run rather than requeuing the delivery, as you are doing.

From java doc

public void commitSync()
Commit offsets returned on the last poll() for all the subscribed list of topics and partitions.
This is a synchronous commits and will block until either the commit succeeds or an unrecoverable error is encountered (in which case it is thrown to the caller).

For commitSync, if a rebalance occurs before the commitSync is called, the commit will fail with an exception.
Hence in your case, the messages will be read again by the new consumer that gets assigned to partitions P1 and P2

One way to get more control over rebalances is to implement ConsumerRebalanceListener which has methods for
a. onPartitionsRevoked
b. onPartitionsAssigned

Advertised hostname+port are deprecated properties, you only need advertised.listeners

For listeners, it should be a socket bind address, such as 0.0.0.0:9094 for all connections on port 9094

When I start the producer it is always trying to connect localhost:9092

Probably because there's a space in your property file before the equals sign (in general, I'd suggest using a yaml config file instead of properties). You can also simply use spring.kafka.bootstrap-servers

Turns out that Flink 1.12 defaults the TimeCharacteristic to EventTime and deprecates the whole TimeCharacteristic flow. So to downgrade to Flink 1.11, you must add the following statement to configure the StreamExecutionEnvironment.