aws-lambda-sqs | Sample AWS Lambda with SQS using Spring Cloud Function | AWS library

We use a Fifo Queue configured with a Lambda Function as a processor. We use MessageGoupId and BatchSize to optimistically remove redundant messages. To rate limit processing we use reserved concurrency. Our function timeout has to be high. The queue maximum receives is set to ten.

When there are a lot of messages in the queue, the Lambda function scales up. Once it scales enough to throttle, the queue processing comes to a complete hold and no further messages are processed until some minutes have passed.

I’m assuming this is because of throttling, because halting always coincides with throttling and when increasing the reserved concurrency, it takes much longer for the processing to come to a halt.

I’m assuming the time until the queue starts up again has to do with the lambda retry limit, the function timeout and the queue visibility timeout. But since I don’t know exactly what happens this is a guess.

There are no errors logged and ultimately all messages are processed, but since processing is time and throughput sensitive, it is not acceptable to have the queue halt for many minutes.

What is happening and how can we solve this? If more information is needed I’d be very happy to debug further.

Edit: Found this: To allow your function time to process each batch of records, set the source queue's visibility timeout to at least 6 times the timeout that you configure on your function. The extra time allows for Lambda to retry if your function execution is throttled while your function is processing a previous batch. and we are certainly in violation, but I'm not sure how / if this explains the observed behaviour.