cron | a cron library for go | Cron Utils library

We have micro service which consumes(subscribes)messages from 50+ RabbitMQ queues.

Producing message for this queue happens in two places

1. The application process when encounter short delayed execution business logic ( like send emails OR notify another service), the application directly sends the message to exchange ( which in turn it is sent to the queue ).

2. When we encounter long/delayed execution business logic We have messages table which has entries of messages which has to be executed after some time.

Now we have cron worker which runs every 10 mins which scans the messages table and pushes the messages to RabbitMQ.

Let's say the messages table has 10,000 messages which will be queued in next cron run,

1. 9.00 AM - Cron worker runs and it queues 10,000 messages to RabbitMQ queue.
2. We do have subscribers which are listening to the queue and start consuming the messages, but due to some issue in the system or 3rd party response time delay it takes each message to complete 1 Min.
3. 9.10 AM - Now cron worker once again runs next 10 Mins and see there are yet 9000+ messages yet to get completed and time is also crossed so once again it pushes 9000+ duplicates messages to Queue.

Note: The subscribers which consumes the messages are idempotent, so there is no issue in duplicate processing

I can have 4 status ( RequiresQueuing, Queued, Completed, Failed )

1. Whenever a message is inserted i can set the status to RequiresQueuing
2. Next when cron worker picks and pushes the messages successfully to Queue i can set it to Queued
3. When subscribers completes it mark the queue status as Completed / Failed.

There is an issue with above logic, let's say RabbitMQ somehow goes down OR in some use we have purge the queue for maintenance.

Now the messages which are marked as Queued is in wrong state, because they have to be once again identified and status needs to be changed manually.

Let say I have RabbitMQ Queue named ( events )

This events queue has 5 subscribers, each subscribers gets 1 message from the queue and post this event using REST API to another micro service ( event-aggregator ). Each API Call usually takes 50ms.

Use Case:

1. Due to high load the numbers events produced becomes 3x.
2. Also the micro service ( event-aggregator ) which accepts the event also became slow in processing, the response time increased from 50ms to 1 Min.
3. Cron workers follows your design mentioned above and queues the message for each min. Now the queue is becoming too large, but i cannot also increase the number of subscribers because the dependent micro service ( event-aggregator ) is also lagging.

Now the question is, If keep sending the messages to events queue, it is just bloating the queue.

https://www.rabbitmq.com/memory.html - While reading this page, i found out that rabbitmq won't even accept the connection if it reaches high watermark fraction (default is 40%). Of course this can be changed, but this requires manual intervention.

So if the queue length increases it affects the rabbitmq memory, that is reason i thought of throttling at producer level.

1. How can i throttle my cron worker to skip that particular run or somehow inspect the queue and identify it already being heavily loaded so don't push the messages ?
2. How can i handle the use cases i said above ? Is there design which solves my problem ? Is anyone faced the same issue ?

Thanks in advance.

Check the accepted answer Comments for the throttling using queueCount

I had a R script configured to run via cron every day at 10am

I have an app with postgres as db, sequelize, and express, and whenever it receives a db query, it just stays there forever, no logging or anything I run postgres in a container which I can connect to through GUI normally When I swapped it for sqlite, it worked perfectly the application

here is the relevant piece of code

I am trying to create a Cron job programmatically in the CloudScheduler Google Cloud Platform using the following API explorer.

Reference: Cloud Scheduler Documentation

Even though I have given the user Owner permission and verified it in Policy Troubleshooter that it has cloudscheduler.jobs.create, I am still getting the following error.

I have a configuration that successfully works and loads cell line data and publishes to various recipients in a cell line topic. It works fine, but when I try to load the JobLauncherTestUtils and JobRepositoryTestUtils, I get an error which says that the JobBuilderFactory is not found. As you will see from my configuration, I do load the JobBuilderFactory and StepBuilderFactory using Lombok which delegates to Spring. Like I said all that works fine but the test Here is the test configuration yaml file

application-test.yml

I've created a task and verified that it exists using SHOW TASKS. I'm now trying to create a subtask using the AFTER parameter of CREATE TASK, but I'm getting the following error: invalid property 'AFTER' for 'TASK'. I can't find any documentation on why this is happening. I think my syntax is correct; it appears to match Snowflake's documentation. This is my code:

Ever since I upgraded my EKS cluster to v1.21, I get the following error when triggering Cronjobs manually:

Experienced issue:

The command * * * * * $(command -v docker) docker run --rm -it --env-file=/home/ubuntu/.env ghcr.io/sebastix/crypto-dca:latest buy 10 ADA won't start. This is tested within the files

• crontab -e
• sudo crontab -e
• sudo nano /etc/crontab

The log E-Mail always prints :

Based on this article, I am trying to generate a repo that runs everytime y push, but also every 4 or 8 hours. as seen by the action history, only one of them was successful, and only on a Mac Machine:

But my goal is to run it on at ubuntu-latest, windows-latest and macos-latest to check the code in all OS. One final goal is to make it so that the message of the scheduled pushs is updated in xxx date. This is my latest code EDITED on sept 23 which can be seen in this repo: