Cheapjack | download manager providing an interface | iOS library

Reading through Weka's StringToWordVector documentation, there seem to be a couple of implementation details different than yours. Here are the top two, based on how likely they are to be the reason for the performance difference you see, in my opinion:

• It seems that by default, the resulting vector is boolean (i.e. noting the existence of a word, rather than number of occurrences)
• If the class attribute is set before vectorizing the text, a separate dictionary is built for each class, then all dictionaries are merged.

While any of them (or other, more minor differences) could be the culprit, my bet is on the second point.

The built-in class allows setting and unsetting each of these options; you could try re-running the 80% version using StringToWordVector with the -C option to use number of occurences rather then a boolean value, and with -O, to use a single dictionary across both classes.

This should allow you to verify whether any of these is indeed the culprit.

EDIT: Regarding the first point, i.e. counting occurences vs. noting word existence (also called Bernoulli and multinomial models), there were several academic papers at the 90s which looked into the differences, e.g. here and here. While usually the multinomial model works better, there are also opposite cases, depending on corpus and classification problem.