p-queue | Promise queue with concurrency control | Reactive Programming library

You probably misunderstood what is returnChannel on the Amqp.outboundGateway and try to rely your logic on it. Please, make yourself familiar with that Publisher Confirms and Returns feature: https://docs.spring.io/spring-amqp/docs/current/reference/html/#cf-pub-conf-ret.

It is also not clear what is a replyBackToHttp flow purpose, but it confuses at the moment with mixed references to other beans.

You probably need to investigate what is a request-reply configuration from Spring AMQP respective and you would probably don't try to use another queue for replies. Although it is still possible: see replyAddress property or RabbitTemplate: https://docs.spring.io/spring-amqp/docs/current/reference/html/#request-reply

From the logs:

It's simple enough to implement it yourself - make an array of functions that, when called, return the Promise. Then implement a limiter function that takes functions from that array and calls them, and once finished, recursively calls the limiter again until the array is empty:

I think that in your scenario the queue approach wouldn't work quite well with Cloud Functions. The queue cannot be implemented in a function as multiple instances won't know about each other, therefore the queue would need to be implemented in some kind of dedicated server, which IMO defeats the purpose of using Cloud Functions as both the queue and the processing could be ran in the same server.

I would suggest having a collection in Firestore that keeps track of the queries that have been requested. This way even if the CSV file isn't still saved on Storage you could check if some function instance is already creating it, then you could sleep the function until the operation completes and return the signed url. Overall the algorithm might look somewhat like this:

I had the same problem, just figured it out. Remove the line:

I found why the above code didn't work, I shouldn't await instance outside of the worker function, but await inside, see below,

numprocs will spawn 8 processes that will then poll the queue every 3 seconds. When daemon is set, these processes will not restart unless told to which has both advantages in terms of server load and disadvantages in the form of some potential edge cases when updating your code base.

Having 8 processes means that you have potentially 8 times the throughput when running jobs.

Example:

There are many scenarios where having multiple processes running in parallel are advantageous.

For instance, say you are processing 1000 users and want to check how many comments each has made in the last month. Say each check takes a minute to process (extreme but it makes a better point), it would take 1000 minutes to complete if you run then in sequence by looping through a array or collection of a 1000 users. That's over 16 hours!

If you queued these as jobs and have numprocs set to 16, then you are done in just over and hour!

If you are not expecting a reply from your message into your foo queue, then you need to tell activeMQ you are not expecting a reply by sending an inOnly type exchange, or it will create a temporary queue and await a reply:

Will it run each of the async functions above in different cpu threads?

No, of course not. That's not what async functions do. All they do is provide nicer syntax for sequential asynchronous code. They're still normal promises with promise callbacks under the hood, running on the same single-threaded event loop as always. See also Is promise.all useful given that javascript is executed in a single thread?

Of course, the got package would be free to use nodejs' worker thread features or something else to do its job, all it needs to do to play nicely with await is return a promise. But that won't affect your asynchronous IIFEs or the p-queue on which the functions are scheduled.