stemmer | Fast Porter stemmer implementation | Natural Language Processing library

The StringToWordVector filter uses the weka.core.DictionaryBuilder class under the hood for the TF/IDF computation.

As long as you create a weka.core.Instance object with the text that you want to have converted, you can do that using the builder's vectorizeInstance(Instance) method.

Edit 1:

Below is an example based on your code (but with Weka classes), which shows how to either use the filter or the DictionaryBuilder for the TF/IDF transformation. Both get serialized, deserialized and re-used as well to demonstrate that these classes are serializable:

1. Is there a practical use case to perform lemmatization on the tokenized data and then stem that lemmatized data or vice versa

2. Does the idea of stemming the lemmatized data or vice versa make any sense theoretically, or is it completely incorrect.

Regarding (1): Lemmatisation and stemming do essentially the same thing: they convert an inflected word form to a canonical form, on the assumption that features expressed through morphology (such as word endings) are not important for the use case. If you are not interested in tense, number, voice, etc, then lemmatising/stemming will reduce the number of distinct word forms you have to deal with (as different variations get folded into one canonical form). So without knowing what you want to do exactly, and whether morphological information is relevant to that task, it's hard to answer.

Lemmatisation is a linguistically motivated procedure. Its output is a valid word in the target language, but with endings etc removed. It is not without information loss, but there are not that many problematic cases. Is does a third person singular auxiliary verb, or the plural of a female deer? Is building a noun, referring to a structure, or a continuous form of the verb to build? What about housing? A casing for an object (such as an engine) or the process of finding shelter for someone?

Stemming is a less resource intense procedure, but as a trade-off it works with approximations only. You will have less precise results, which might not matter too much in an application such as information retrieval, but if you are at all interested in meaning, then it is probably too coarse a tool. Its output also will not be a word, but a 'stem', basically a character string roughly related to those you get when stemming similar words.

Re (2): no, it doesn't make any sense. Both procedures attempt the same task (normalising inflected words) in different ways, and once you have lemmatised, stemming is pointless. And if you stem first, you generally do not end up with valid words, so lemmatisation would not work anyway.

I am not really familiar with the model you use and might be mising something here, but it might be that your test set is not representative of the data. Perhaps there is something in the 2021 data that causes it to be easier to predict.

You might want to try something like sklearn's train_test_split() with shuffle=True to ensure the test set is a representative random subset of the data and see if you get more balanced performances between the sets this way.

Depending on which task exactly you are doing, 300 entries is really not a lot for a test set in NLP, so that small test set size alone might distort the test results.

It is a bit difficult to give advise on how to generally improve the predictions without knowing what you generally are trying to do. I assume it has to do with doing some kind of two class classification on stemmed tokens?

Can you clarify/give an example for an entry and the desired predictions?

You should use the json module to load in JSON data as opposed to reading in the file line-by-line. Whatever procedure you build yourself is likely to be fragile and less efficient.

Here is the looping structure that you're looking for:

• data is not defined, your json data is loaded in trainer_load.

I was watching this tutorial. I copied over the stuff he wrote, but with some changes in the variables and other stuff. Then I got the error (The error is below).

• Change in the variable is the cause of the error. You must change the variable in every occurrence if you want to change, you just changed while defining the variable
• In the link you have refered the code is:

(Asking for software/data recommendations is off-topic for StackOverflow; but I have tried to give a more general "approach" answer.)

1. Another approach to finding related words would be one of the machine learning approaches. If you are dealing with words in isolation, look at word embeddings such as GloVe or Word2Vec. Spacy and gensim have libraries for working with them, though I'm also getting some search hits for tutorials of working with them in nltk.

2/3. One of the (in my opinion) core reasons for the success of Princeton WordNet was the liberal license they used. That means you can branch the project, add your extra data, and redistribute.

You might also find something useful at http://globalwordnet.org/resources/global-wordnet-grid/ Obviously most of them are not for English, but there are a few multilingual ones in there, that might be worth evaluating?

Another approach would be to create a wrapper function. It first searches a lookup list of fixes and additions you think should be in there. If not found then it searches WordNet as normal. This allows you to add 'succeed', 'success', 'successful', and then other sets of words as end users point out something missing.

wordStem does not employ a dictionary but uses grammatical rules to do stemming (which is a rather crude approximation to lemmatisation btw). Here is an example:

This query first group by names then groups each name's values. By setting sizes, you can arrange number of facets you want and number of items in each facet. I think it does what you need.

Note that if you have too many documents and if performance matters, this query may perform bad.