mongo-deep-mapreduce | Use Hadoop MapReduce directly on Mongo data

mongo-deep-mapreduce is a Java library typically used in MongoDB applications. mongo-deep-mapreduce has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub, Maven.

This is a library of MongoDB related Hadoop MapReduce classes, in particular an InputFormat that reads directly from Mongo’s binary on-disk format. Developed by Peter Bakkum at Groupon in Palo Alto. Problem: If you want to use Hadoop MapReduce with a Mongo collection you currently have two options: - You can execute one or more cursors over the entire cluster in your MapReduce job. - You can export the collection as BSON or JSON, which also executes a cursor over the entire collection, and MapReduce over the exported data. However, with a large data set that significantly exceeds the available memory on the Mongo host, these options can both be prohibitively time consuming. Solution: Move the raw Mongo files into HDFS, without exporting, and MapReduce over them using this library. Mongo uses a proprietary binary format to manage its data, which is essentially a doubly-linked list of BSON records. By reading this format directly, we obviate the need for expensive data conversion prior to a Hadoop MapReduce, and we can utilize the full throughput of the Hadoop cluster when reading the data, rather than using single-threaded cursors.