datasketch | LSH Forest , Weighted MinHash | Hashing library

You use basic authentication. You should just be able to send your query to druid with an URL like this:

I resolved the issue by changing the hdp.version in the mapred-site.xml manually. I was getting following exception in middleManager.log

java.lang.IllegalArgumentException: Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path

But Still the segment metadata is showing Request failed with status code 404.

RDD`s to the rescue

LSH doesn't bucket entire documents, nor does it bucket individual minhashes. Rather, it buckets 'bands' of minhashes.

LSH is a means of both reducing the number of hashes stored per document, and reducing the number of hits found when using these hashes to search for similar documents. It achieves this by combining multiple minhashes together into a single hash. So, for example, instead of storing 200 minhashes per document, you might combine them in bands of four to yield 50 locality sensitive hashes.

The hash for each band is calculated from its constituent minhashes using a cheap hash function such as FNV-1a. This loses some information, which is why LSH is said to reduce the dimensionality of the data. The resulting hash is the bucket.

So the bucket for each band of minhashes within a document is calculated without requiring knowledge of any other bands or any other documents.

Using LSH hashes to find similar documents is simple: Let's say you want to find documents similar to document A. First generate the (e.g.) 50 LSH hashes for document A. Then look in your hash dictionary for all other documents that share one or more of these hashes. The more hashes they share, the higher their estimated jaccard similarity (though this is not a linear relationship, as it is when using plain minhashes).

The fewer total hashes stored per document, the greater the error in estimated jaccard similarity, and the greater the chance of missing similar documents.

Here's a good explanation of LSH.

I have the impression that problem lies to the underlying implementation of the logger.

I am not an expert on logging configurations but I think specifying the properties file through -Djava.util.logging.config.file does not have any effect since the logger that is used (according to the classpath you provided) is the Log4J implementation (slf4j-log4j12-1.7.25.jar) and not the one of the jdk (https://mvnrepository.com/artifact/org.slf4j/slf4j-jdk14/1.7.26).

I think that the right property to use for the log4j implementation is the folowing: -Dlog4j.configuration=file:C:\Users\user0\workspaces\apache-projects\apache-calcite\core\src\test\resources\log4j.properties

It happened because the druid-kafka-indexing-service extension was missing from the extension list of common.runtime.properties.

I don't think it might be possible to save python objects in DataFrames, but you can circumvent this in a couple of ways:

• Store the result instead of the object (not sure about how MinHash works, but if the value is numerical/string, it should be easy to extract it from the class object).

• If that is not feasible because you still need some properties of the object, you might want to serialize it using Pickle, saving the serialized result as an encoded string. This forces you to de-serialize every time that you want to use the object.

  final_df_limit.rdd.map(lambda x: base64.encodestring(pickle.dumps(CalculateMinHash(x),))).toDF()

• An alternative might be to use the Spark MinHash implementation instead, but that might not suit all your requirements.

It turned out I was hammering the filesystem too hard, making too many concurrent read requests for files on the same server. That server could only allow a fixed number of reads in a given period, so any requests over that limit received a Connection Timed Out response.

The solution was to wrap each file read request in a while loop. Inside that while loop, try to read the appropriate file from disk. If the Connection timed out error springs, sleep for a second and try again. Only once the file has been read may the while loop be broken.