retryable | Python Retry Decorator | Functional Programming library

When debugging, we do see concurrently when this transaction is being run, documents returned in the cts:search are locked for updates in other transactions. We are well aware of this possibility and are okay with it.

You may think that you are okay with it, but you are running into performance issues that are likely due to it, and are looking to avoid timeouts - so you probably aren't okay with it.

When you perform a search in an update transaction, all of the fragments will get a read-lock. You can have multiple transactions all obtain read locks on the same URI without a problem. However, if one of those transactions then decides it wants to update one of those documents, it needs to promote it's shared read-lock to an exclusive write-lock. When that happens, all of those other transactions that had a read-lock on that URI will get restarted. If they need access to that URI that has an exclusive write-lock then they will have to wait until the transaction that has the write-lock completes and lets go.

So, if you have a lot of competing transactions all performing searches with the same criteria and trying to snag the first item (or first set of items) from the search results, they can cause each other to keep restarting and/or waiting, which takes time. Adding more threads in an attempt to do more makes it even worse.

There are several strategies that you can use to avoid this lock contention.

Instead of cts:search() to search and retrieve the documents, you could use cts:uris(), and then before reading the doc with fn:doc() (which would first obtain a read-lock) before attempting to UPSERT (which would promote the read-lock to a write-lock), you could use xdmp:lock-for-update() on the URI to obtain an exclusive write-lock and then read the doc with fn:doc().

If you are trying to perform some sort of batch processing, using a tool such as CoRB to first query for the set of URIs to process (lock-free) in a read-only transaction, and then fire off lots of worker transactions to process each URI separately where it reads/locks the doc without any contention.

You could also separate the search and update work, using xdmp:invoke-function() or xdmp:spawn-function() so that the search is executed lock-free and the update work is isolated.

Some resources that describe locks and performance issues caused by lock contention:

• https://www.marklogic.com/blog/resolving-unresolvable-deadlocks/
• https://help.marklogic.com/Knowledgebase/Article/View/17/0/understanding-xdmp-deadlock
• https://help.marklogic.com/Knowledgebase/Article/View/understanding-locking-in-marklogic-using-examples
• https://help.marklogic.com/Knowledgebase/Article/View/strategies-to-ensure-if-locking-is-the-root-cause-for-performance-degradation-in-marklogic

According to the docs, the endpoint url is:

savingsplans.amazonaws.com

You can manually specify correct endpoint:

That is a good suggestion; it probably should be the default behavior (or at least optionally).

Please open a feature request on GitHub.

There is a, somewhat, related discussion here: https://github.com/spring-projects/spring-kafka/discussions/2101

I was able to solve this issue with the help of an AWS rep. It turns out I needed a public and private subnet in my VPC containing the lambda. The lambda itself had to be in a private subnet with the public subnet containing a NAT gateway and an internet gateway. Instead of a single route table in the VPC, I needed separate route tables for the two subnets. The private one contains the peering connection route and VPC CIDR route like I mentioned in my question but also contains a route with destination 0.0.0.0/0 with the NAT gateway as the target. The public subnet route table contains the VPC CIDR route as well as a route with destination 0.0.0.0/0 with the internet gateway as the target.

You'll need to get the access and secret key from AWS and insert them in place of the XXXXXXX placeholders.

You can get this information in the AWS Cloud -> IAM -> Access Management -> Users -> Select your user -> Security credentials -> Access Keys

You will find here the Access Key ID, but the Secret Key is only shown once when you are creating this item. You maybe have it stored somewhere, or you can create another Access Key pair and use that.

I have done this and I can connect to AWS Toolkit fine.

AWS Toolkit Config

What you really need is something like:

You have to annotate the Keycloak with @Retryable.

According to the documentation:

ConditionalCheckFailedException Message: The conditional request failed.

You specified a condition that was evaluated to be false. For example, you might have tried to perform a conditional update on an item, but the actual value of the attribute did not match the expected value in the condition.

OK to retry? No

So the problem is with the ConditionExpression, the code attribute does not exist in some records or it exists with undefined value. try to change attribute_exists with attribute_type:

It looks like the cause was embarrassingly simple. Some sort of policy or cleanup process removed dev.azure.com from the list of trusted sites from our servers. Once I added it back to this, it seems to work.