minhash | Quickly estimate the similarity between many sets | Hashing library

Was able to solve the issue by upgrading the cluster to c5 2x large vs m4 large.

the column from df2 will not be known to df1 unless you join them and create one object, you can try to first crossjoin both and then try your code:

Package author here. Yes, it would be wasteful to use more hashes/bands than you need. (Though keep in mind we are talking about kilobytes here, which could be much smaller than the original documents.)

The question is, what do you need? If you need to find only matches that are close to identical (i.e., with a Jaccard score close to 1.0), then you don't need a particularly sensitive search. If, however, you need to reliable detect potential matches that only share a partial overlap (i.e., with a Jaccard score that is closer to 0), then you need more hashes/bands.

Since you've read MMD, you can look up the equation there. But there are two functions in the package, documented here, which can help you calculate how many hashes/bands you need. lsh_threshold() will calculate the threshold Jaccard score that will be detected; while lsh_probability() will tell you how likely it is that a pair of documents with a given Jaccard score will be detected. Play around with those two functions until you get the number of hashes/bands that is optimal for your search problem.

Here are some things that you should double-check as they are likely culprits:

• when you create your mapping you should change from "name" to "text" in your client.indices.create method inside body param, because your json document has a field called text: