rabbitmq | RabbitMQ Dockerfile for trusted automated Docker builds | Continuous Deployment library

I have a web application that publishes messages to a topic then several Windows services that subscribe to those topics, some with multiple instances. If the services are running when the messages are published everything works correctly but if they are not then the messages are retained on the queue(s) subscribing to that topic but aren't read when the services start back up.

The desired behavior-

1. When a message is published to the topic string MyTopic, it is read from the MyTopicQueue only once. I use some wildcard topics so each message is sent to multiple queues, but multiple instances of a services subscribe to the same topic string and each message should be read by only of those instances

2. If the subscribers to the MyTopic topic aren't online when the message is published then the messages are retained on MyTopicQueue.

3. When the Windows services subscribing to a particularly topic come back on line each retained message is read from MyTopicQueue by only a single subscriber.

I've found some [typically for IBM] spotty documentation about the MQSUBRQ and MQSO_PUBLICATIONS_ON_REQUEST options but I'm not sure how I should set them. Can someone please help figure out what I need to do to get my desired behavior? [Other than switching back to RabbitMQ which I can't do though I'd prefer it.]

My options:

I'm working with Symfony 4.4 and Symfony Messenger

Messenger configuration includes a transport and routing:

We have micro service which consumes(subscribes)messages from 50+ RabbitMQ queues.

Producing message for this queue happens in two places

1. The application process when encounter short delayed execution business logic ( like send emails OR notify another service), the application directly sends the message to exchange ( which in turn it is sent to the queue ).

2. When we encounter long/delayed execution business logic We have messages table which has entries of messages which has to be executed after some time.

Now we have cron worker which runs every 10 mins which scans the messages table and pushes the messages to RabbitMQ.

Let's say the messages table has 10,000 messages which will be queued in next cron run,

1. 9.00 AM - Cron worker runs and it queues 10,000 messages to RabbitMQ queue.
2. We do have subscribers which are listening to the queue and start consuming the messages, but due to some issue in the system or 3rd party response time delay it takes each message to complete 1 Min.
3. 9.10 AM - Now cron worker once again runs next 10 Mins and see there are yet 9000+ messages yet to get completed and time is also crossed so once again it pushes 9000+ duplicates messages to Queue.

Note: The subscribers which consumes the messages are idempotent, so there is no issue in duplicate processing

I can have 4 status ( RequiresQueuing, Queued, Completed, Failed )

1. Whenever a message is inserted i can set the status to RequiresQueuing
2. Next when cron worker picks and pushes the messages successfully to Queue i can set it to Queued
3. When subscribers completes it mark the queue status as Completed / Failed.

There is an issue with above logic, let's say RabbitMQ somehow goes down OR in some use we have purge the queue for maintenance.

Now the messages which are marked as Queued is in wrong state, because they have to be once again identified and status needs to be changed manually.

Let say I have RabbitMQ Queue named ( events )

This events queue has 5 subscribers, each subscribers gets 1 message from the queue and post this event using REST API to another micro service ( event-aggregator ). Each API Call usually takes 50ms.

Use Case:

1. Due to high load the numbers events produced becomes 3x.
2. Also the micro service ( event-aggregator ) which accepts the event also became slow in processing, the response time increased from 50ms to 1 Min.
3. Cron workers follows your design mentioned above and queues the message for each min. Now the queue is becoming too large, but i cannot also increase the number of subscribers because the dependent micro service ( event-aggregator ) is also lagging.

Now the question is, If keep sending the messages to events queue, it is just bloating the queue.

https://www.rabbitmq.com/memory.html - While reading this page, i found out that rabbitmq won't even accept the connection if it reaches high watermark fraction (default is 40%). Of course this can be changed, but this requires manual intervention.

So if the queue length increases it affects the rabbitmq memory, that is reason i thought of throttling at producer level.

1. How can i throttle my cron worker to skip that particular run or somehow inspect the queue and identify it already being heavily loaded so don't push the messages ?
2. How can i handle the use cases i said above ? Is there design which solves my problem ? Is anyone faced the same issue ?

Thanks in advance.

Check the accepted answer Comments for the throttling using queueCount

I am very new to Elixir. I have built an app which runs locally, which works fine. But now i need to build a container for it with Docker.

However, every attempt to do a release seems to try and connection to RabbitMQ (which is running locally, as a Docker Container).

I don't want, cant have it try and connect to Rabbit each time this container is built, as it will need to be built by a CI / CD pipeline and will never have access to any Rabbit. I have set it up with an ENV, but this needs to be set within my YAML when deploying to my k8s cluster.

So this is the Dockerfile:

I am trying to get my deployment to only deploy replicas to nodes that aren't running rabbitmq (this is working) and also doesn't already have the pod I am deploying (not working).

I can't seem to get this to work. For example, if I have 3 nodes (2 with label of app.kubernetes.io/part-of=rabbitmq) then all 2 replicas get deployed to the remaining node. It is like the deployments aren't taking into account their own pods it creates in determining anti-affinity. My desired state is for it to only deploy 1 pod and the other one should not get scheduled.

Celery disconnects from RabbitMQ each time a task is passed to rabbitMQ, however the task does eventually succeed:

My questions are:

1. How can I solve this issue?
2. What improvements can you suggest for my celery/rabbitmq configuration?

Celery version: 5.1.2 RabbitMQ version: 3.9.0 Erlang version: 24.0.4

RabbitMQ error (sorry for the length of the log:

I need to copy date from one source (in parallel) to another with batches.

I did this:

We have set our Kafka Connect to be able to read credentials from a file, instead of giving them directly in connector config. This is how a login part of connector config looks like:

"connection.user": "${file:/kafka/pass.properties:username}",

"connection.password": "${file:/kafka/pass.properties:password}",

We also added these 2 lines to "connect-distributed.properties" file:

config.providers=file

config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider

Mind that it works perfectly for JDBC connectors, so there is no problem with the pass.properties file. But for other connectors, such as couchbase, rabbitmq, s3 etc. it causes problems. All these connectors work fine when we give credentials directly but when we try to make Connect to read them from a file it gives some errors. What could be the reason? I don't see any JDBC specific configuration here.

EDIT:

An error about couchbase in connect.log:

I use SignalR in order to expose RabbitMQ messages to browsers. This works fine with one app instance obviously. The question is if it could work with multiple instances too without a backplane. I understand that SignalR client could be disconnected from the pod A and connected back to the pod B but what exactly is the issue here? I am fine to lose some messages during reconnection. Is it the only issue? Is reconnection to the pod B treated as a regular new connection so that the client is just subscribed again as it was subscribed normally without reconnection? Or the system doesn't have input parameters it had during initial subscription and therefore it cannot resubscribe without hints?