MITIE | MITIE: library and tools for information extraction | Natural Language Processing library

Setting things up:

For starters, you can download the English Language Model which contains Corpus of annotated text from a huge dump in a file called total_word_feature_extractor.dat.

After that, download/clone the MITIE-Master Project from their official Git.

If you are running Windows O.S then download CMake.

If you are running a x64 based Windows O.S, then install Visual Studio 2015 Community edition for the C++ compiler.

After downloading, the above, extract all of them into a folder.

Open Developer Command Prompt for VS 2015 from Start > All Apps > Visual Studio, and navigate to the tools folder, you will see 5 sub-folders inside.

The next step is to build ner_conll, ner_stream, train_freebase_relation_detector and wordrep packages, by using following Cmake commands in the Visual Studio Developer Command Prompt.

Something like this:

For ner_conll:

In your old version you should be able do a request like this (documented here):

For UNIX O.S:

pip install git+https://github.com/mit-nlp/MITIE.git

For Windows O.S:

I solved my issue by doing the following, hope it helps someone in the future.

1) Firstly clone the git package from MITIE's official GIT page.

2) After downloading, Seems like ~\MITIE-master\mitielib has a __init__.py file which makes the directory a valid Python module. Navigate to the ~\MITIE-master\mitielib folder, it will look something like this:

3) Packages installed from pip reside in the C:\Anaconda3\Lib\site-packages\ directory. Make a new folder called mitie and paste the contents there.

4) Lastly modify your configuration file as follows, in the mitie_file key's value provide the path to the total_word_feature_extractor.dat file:

You have to explicitly terminate token array with NULL:

I've been looking for something like this, too. Simply for a "generic" (and hence not very useful) NLU backend. The only thing I've found so far is a trained model with 9 news categories (not very generic). See blog post here: http://eric-yuan.me/ner_1/

If you have the option to switch NERs, spaCy has a trained model available by default. Its visualisation front end can be found by google "displacy"

If you find anything else, let me know!

EDIT: Spent the day looking into this and I think I've found what you're after. If you go to https://github.com/mit-nlp/MITIE/releases there you'll find MITIE's own NER model trained on Wikipedia, Freebase, etc. The actual training dataset is there too. The README on their github page provides example on how to use the pre-trained model. You can also investigate the ner.py file in the examples folder to see how to use the pre-trained model in python code.

If you poke around deep enough in the MITIE repo's on Github you can find your answer. For example here is a bit of information about what goes into that file.

As for what's inside, yes, it's a variant of word2vec based on the two step CCA method from this paper: http://icml.cc/2012/papers/763.pdf. I also upgraded it to include something that is similar to the CCA method but works on out of sample words by analyzing their morphology to produce a word vector. This significantly improved the results on datasets containing lots of words not in the original dictionary.

As far as how MITIE integrates into Rasa, it is one of a few backend choices for Rasa. It provides a few pipeline components that can do both intent classification and NER. Both of which use an SVM and use the total_word_feature_extractor.dat to provide the individual word vectors.