ibleu | interactive scoring environment for machine translation | Translation library

While working on my dissertation on statistical machine translation, I had to examine translation output for quality again and again. The way I used to do it was to open the source XML in one terminal window, the hypothesis XML system in another, the references in yet another window and the document/segment BLEU scores in a fourth window. I would then try to find documents with segments that had anomalously high/low BLEU scores and then copy paste the source into my browser to see how Google translated it. Things would be even worse if I was comparing my system to a baseline system. This tool allows you to drill down into documents and perform this sort of qualitative examination and comparison in a very visual and interactive manner. It's not perfect by any means but I think it's interesting and probably useful to many people in the community. In the last five years, every major browser maker (Mozilla, Apple and Google) has invested a lot of effort into heavily optimizing its specific implementation of Javascript interpreters/compilers. Javascript, the language itself, has also matured and has gained a very useful set of third-party libraries. On top of that, with the advent of HTML5 and CSS2/CSS3, it has become almost trivial to do interesting visualizations in the browser. These thoughts, coupled with the fact that everyone (and, I mean, everyone) has a browser, made me curious enough to try and create this tool. It's still the most frequently used metric in the machine translation community and probably the simplest to implement. This tool contains a pure Javascript implementation of the BLEU metric (based on the NIST mteval script v13a). Porting BLEU to Javascript was really pretty easy given how mature the language has become. In short: the BLEU score for a set of translations translation is defined as the product of two numbers --- an n-gram precision score (what percentage of the n-grams or phrases from the translations also appear in the reference translations) and a brevity penalty (a number indicating if the translations are too short compared to the reference, a brevity penalty of 1.0 means that the lengths of the translations and the references are quite close to each other). For example, if the n-gram precision score is 0.0442 and the brevity penalty is 0.8356, the final BLEU score is 0.0442 x 0.8356 = 0.0370. In machine translation literature, the convention is to report BLEU scores as percentages. Therefore, iBLEU reports 0.0370 as 3.70. Good quality translations will have BLEU scores closer to 1.0 (or 100.0 in iBLEU). Generally, BLEU scores against a single reference translation are lower than if you have multiple reference translations since then the translation has a better chance of matching some of the n-grams. However, if you want to use multiple reference translations, you need to use the XML format since there is no way to specify multiple reference translations using TXT files. For more details, please refer to the BLEU Wikipedia page. The format is the official NIST XML format as described in this DTD. Please make sure that the XML files you are using are valid XML and compliant with this DTD. This DTD is also included with the official release download zip file.