opennlp | different packages | Natural Language Processing library

I solved the problem. This error was being thrown simply because the script could not be found and I had to change this line from: RutaEngine.PARAM_MAIN_SCRIPT, "myscript.ruta" to: RutaEngine.PARAM_MAIN_SCRIPT, "myscript"

However, I did a few other things before this that may have contributed to the solution so I am listing them here:

1. I added the ruta nature to my eclipse project
2. I moved the myscript from resources to a script package

Yes, the default tokenizer is OpenNLP and it works with no configuration needed. It will guess the language if you don't set it, but if you know the document language it is better to use set_language (and language=...) in queries, since language detection is unreliable on short text.

However, OpenNLP tokenization (not detecting) only supports Danish, Dutch, Finnish, French, German, Hungarian, Irish, Italian, Norwegian, Portugese, Romanian, Russian, Spanish, Swedish, Turkish and English (where we use kstem instead). So, no CJK.

To support CJK you need to plug in your own tokenizer as described in the linguistics doc, or else use ngram instead of tokenization, see https://docs.vespa.ai/documentation/reference/schema-reference.html#gram

n-gram is often a good choice with Vespa because it doesn't suffer from the recall problems of CJK tokenization, and by using a ranking model which incorporates proximity (such as e.g nativeRank) you'l still get good relevancy.

Ok, it's normal you have this message because in your code, you' ll do this :

After a lot of research I stumbled on someone with the same problem using NLTK. They were advised to "hack" NLTK by adding a pronoun like "they" before the command to force the parser to see the input as a verb phrase. So I would give OpenNLP "they open door" and get back (S (NP (PRP they)) (VP (VBP open) (NP (NN door)))), at which point I can just extract the verb phrase.

It's certainly not ideal! But for now it will work for my requirements.

I've found the solution. The LemmatizerTrainerME is inside opennlp tools jar file. So that's what I did:

I ran Windows Powershell inside lib folder with the following command: opennlp opennlp-tools-1.9.3.jar LemmatizerTrainerME -model en-lemmatizer.bin -lang en -data /path/to/en-lemmatizer.dict -encoding UTF-8 and it worked.

TLDR: I ran Powershell inside the folder that contains opennlp tools and added the tools file name before the arguments so it could access LemmatizerTrainerME

Well, the answer to this one did not come from the direction in which the question was asked. It turns out that there was a code sample in the OpenNLP documentation that was wrong, and no amount of parameter tuning would have solved it. I've submitted a jira to the project so it should be resolved; but for those who make their way here before then, here's the rundown:

Documentation (wrong):

You have the line ${java.home}/lib/rt.jar in your configuration for proguard. This is no longer valid in JDK11 as it was removed in that version of Java.

SOlr requires different features that require an optional libraries. All of these are comes with Solr. You need to adjust solr.install.dir like already mentioned in file named INSTALL.md

Updating path to solr.install.dir=/opt/solr in solrcore.properties to fix the issue.

Check the jar named as "icu4j-62.1.jar". Check the path of the same is mentioned in solrConfig.xml and check it the lib is getting loaded.