deduplication | Fast multi-threaded content | Stream Processing library

I'd recommend reviewing Levenstein distance as this is a common algorithm to identify similar strings. Library FuzzWuzzy(goofy name I know) implements it with 3 different approaches. See this article for more info

Here's a starting place that compares each string against every other string. You mention having different thresholds, so all would need to do is loop through l_match and group them depending on your desired thresholds

String interning happens only for constant strings and strings that you manually intern.

String deduplication happens automatically in the background for all strings, including ones you create at runtime.

Are they deleted?

Yes, they are accepted by the queue & then deleted by SQS.

The purpose of the deduplication ID is to prevent duplicate messages from being consumed & to guarantee "exactly once" delivery.

My suggestion would be as follows. You only add those texts with a score less than (or equal) 0.9.

It EventBridge (EB) has at-least-once deliver which means you can get more then one event of the same type. But if this is not an issue, and your only concern is SQS, then yes, EB supports SQS FIFO targets:

EventBridge lets you set a variety of targets—such as Amazon SQS standard and FIFO queues—which receive events in JSON format.

You can do it all quite easily with Miller, which is available here as a static binary. Put the mlr executable somewhere in your PATH and you're done with the installation.

For starters, I'll assume that we're working with two CSV files with no inconsistency in the column names:

Unfortunately (?), the easiest way to know for sure what those categories map to is to look at OpenJDK source code. The NMT tag you are looking for is mtServiceability. This would show that "serviceability" are basically diagnostic interfaces in JDK/JVM: JVMTI, heap dumps, etc.

But the same kind of thing is clear from observing that stack trace sample you are showing mentions ThreadStackTrace::dump_stack_at_safepoint -- that is something that dumps the thread information, for example for jstack, heap dump, etc. If you have a suspicion for the memory leak in that code, you might try to build a MCVE demonstrating it, and submitting the bug against OpenJDK, or showing it to a fellow OpenJDK developer. You probably know better what your application is doing to cause thread dumps, focus there.

That being said, I don't see any obvious memory leaks in StackFrameInfo, neither can I reproduce any leak with stress tests, so maybe what you are seeing is "just" thread dumping over the larger and larger thread stacks. Or you capture it when thread dump is happening. Or... It is hard to say without the MCVE.

Update: After playing with MCVE, I realized that it reproduces with 17.0.1, but not with either mainline development JDK, or JDK 18 EA, or JDK 17.0.2 EA. I tested with 17.0.2 EA before, so was not seeing it, dang. Bisection between 17.0.1 and 17.0.2 EA shows it was fixed with JDK-8273902 backport. 17.0.2 releases this week, so the bug should disappear after you upgrade.

after a search, and trying duplicating messages with different message system attributes, i found that indeed :

sqs does consider message attributes and message system attributes as "Content"

This might help: The message group ID is used for ordering of SQS messages in groups.

So, if you have multiple group IDs like 1 and 2, then within those groups, the SQS messages will always be in order. The ordering will be preserved relative to the group and not the queue as a whole.

This can be used in cases like, if you are sending out events for different user details updates, you would want to group the data for a particular user in order, and at the same time you don't care about the ordering of the queue as a whole. You can then set the group ID as the unique user ID in this case.

If there is no use-case of creating groups, and there is only one group and you want the message order to be preserved as a whole, just use the same groupID for all messages.

more info here: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html