StringTokenizer | benchmarking variations and for eventual inclusion

It's the Combiner's fault. You are specifying in the driver class that you want to use MR2Reducer both as a Combiner and a Reducer in the following commands:

I didn't check all the code, but there is an error in Ellipse class, check my remarks

You may use this regex for your customised word matches:

For solving this issue. Read Documantation

In this case is think you should use `

`Integer.parseInt(input);

Things are surprisingly simple in this use case of Hadoop's MapReduce, because Hadoop tends to read the input documents line-by-line even with FileInputFormat being explicitly specified for the format of the input data of a MR job (this goes far beyond the scope of your question, but you can check out about map and file splits in Hadoop here and here).

Since each mapper instance is going to have a line as its input, the only thing you have to worry about is:

1. splitting the text into words (after cleaning them up from punctuation, loose spaces, turning all of them to lowercase, etc.),

2. getting rid of the duplicates to end up with just the unique words of said line,

3. and signing every unique word as key with 1 as value, classic WordCount style.

For 2. you can use a HashSet which (as you'd expect) is a Java data structure that only keeps unique elements while ignoring duplicates, to load every token to it and then iterate it to write the key-value pairs and send them to the reducer instances.

This type of application can look like this (I changed the way you tokenize the text in the Map function because it didn't seem to split each word but just split between punctuation symbols):

You are indeed right, you do need two MapReduce jobs chained together. More specifically, you need:

• one job to compute the wordcount for every word stored inside the input documents,

• and one job to be able to "sort" all those words and wordcounts in order to pick and output the top N of them.

The first job is fairly similar to what you have come up with already, so I'm going to focus on the second job, to make things a bit clearer about how TopN works in the MapReduce paradigm.

Thinking of the TopN MR job as a standalone thing, we know that this particular job will receive a bunch of key-value pairs where every word from the last step is going to be the key and its wordcount is going to be the value. Since mappers and reducers are isolated instances of the map and reduce functions running in parallel, we need to find a way to first find the TopN words locally (i.e. for every mapper), and then group all of those local TopN results to find the "global" TopN words for all the data given to the application by the input.

So, the TopNMapper will first have to create a TreeMap (a Java key-value data structure that internally sorts its elements by key) in the setup function (so before mapper instances will be created), which every mapper will initialize an object of it and put every word and its wordcount as elements. For this type of computation (TopN), we will put the wordcount as key and the word as value to have a ascending order sorted list of the words. Since we only need to find out the top N of the words here, it is safe to say that we only want the top N words for every mapper, so we can remove all of the other elements below and have a TreeMap of N elements, which will be given to the reducers in the end of the mappers' execution (i.e. through the cleanup function). The mappers will write key-value pairs where the words are going to be keys and their wordcounts are going to be values, like that:

Now for the TopNReducer, we need to do the same exact thing using a TreeMap data structure again to populate it with all of the local TopN elements, remove the elements that are not the top N of them, and write the words and their wordcounts as output. To be more "clean" with the approach, we can "reverse" words and wordcounts in the key-value pair structure so we can have the wordcounts as key and the words as value. This results into an (ascending order) sorted amount of key-value pairs that are going to be stored in disk after this job is done like this:

wordcount, word>

The program that can do such a thing in 2 MR jobs looks like the following (where we set N as a global Configuration value inside the main function with the conf.set("N", "10"); command, and access it in the setup functions of the TopNMapper and TopNReducer classes), with all of the classes being put in one class TopNWordCount for simplicity:

I think your code is fine but is failing to perform well as per the evaluators. That means you need to change algorithm. Have you tried other strategies like DFS?

Even though your Java-based logic here looks sound, Map and Reduce function implemented functions in Hadoop are more myopic or independent than one thinks. To be more precise, you declare public static variables in the parent class and initialize them in the driver/main function, but the mapper/reducer instances do not have any access to the driver, but only to their strict scopes within the TokenizerMapper and IntSumReducer classes. This is why n and k seem empty when you look inside the mappers and reducers.

Since your program has only one job and is executed within a single Hadoop Configuration, no Hadoop Counters are needed here. You can just declare Configuration based values that will be accessed by every mapper and reducer right before the executions of the Map and Reduce functions, through the setup functions inside the TokenizerMapper and IntSumReducer classes.

To declare those types of values in order to pass them to the MapReduce functions, you can just do something like this in the driver/main method:

conf.set("n", args[2]);

And then access this value (while converting it fron String to int) in the setup methods of TokenizerMapper and IntSumReducer:

n = Integer.parseInt(context.getConfiguration().get("n"));

So the program can look like this:

I suggest you change your arrays so that they contain integer numbers and not strings. Then you can use a recursive solution: