fasttext | fasttext with hierarchical softmax | Natural Language Processing library

From looking at the _FastText Python model class in Facebook's source...

https://github.com/facebookresearch/fastText/blob/a20c0d27cd0ee88a25ea0433b7f03038cd728459/python/fasttext_module/fasttext/FastText.py#L99

...it looks like, at least when creating a model, all the hyperparameters are added as attributes on the object.

Have you checked if that's the case on your loaded model? For example, does ft.dim report 300, and other parameters like ft.minCount report anything interesting?

Update: As that didn't seem to work, it also looks like the _FastText model wraps an internal instance of a native (not-in-Python) FastText model in its .f attribute. (See a few lines up from the source code I pointed to earlier.)

And that native-instance is set up by the module specified by fasttext_pybind.cc. That code looks like it specified a bunch of read-write class variable, associated with the metaparameters - see for example starting at:

https://github.com/facebookresearch/fastText/blob/a20c0d27cd0ee88a25ea0433b7f03038cd728459/python/fasttext_module/fasttext/pybind/fasttext_pybind.cc#L88

So: does ft.f.minCount or ft.f.dim return anything useful from a post-loaded model ft?

For information about the difference between .bin and .vec files, you can read this question.

In any case, MUSE expects .vec files.

If you want to convert a .bin file to a .vec file, this answer will probably help you.

First, you must decide if you are interested in ortographic similarity or semantic similarity.

In this case, you score the distance between two strings. There are various metrics for computing edit distance. Levenshtein distance is the most common: you can find various python implementations, like this.

"gold" is similar to "good", but not similar to "metal".

In this case, you measure how much two strings have a similar meaning.

fastText and other word embeddings fall into this case, even if they also take into account ortographic aspects.

"gold" is more similar to "metal" than to "good".

If you have a limited number of words in your list, you can use an existing word embedding, pretrained on your language. Based on this word embedding, you can compute the word vector for each word/sentence in your list, then compare the vector for your new word with the vectors from the list, using cosine similarity.

FastText's advantage over word2vec or glove for example is that they use subword information to return vectors for OOV (out-of-vocabulary) words.

So they offer two types of pretrained models : .vec and .bin.

.vec is a dictionary of word -> vector information, the word vectors are pre-computed for the words in the training vocabulary.

.bin is a binary fasttext model that can be loaded using fasttext.load_model('file.bin') and that can provide word vector for unseen words (OOV), can be trained more.

In your case you are loading a .vec file so vectors is the "final form" of the data, fasttext.load_model expects a .bin file name. If you need more than a python dictionary you can use gensim.models.keyedvector.

Since trained word vectors are independent from the way they were trained (Word2Vec, FastText, VarEmbed etc), they can be represented by a standalone structure, as implemented in this module.

Valid json format of test.json

For fasttext v0.9.2 this can be solved by adding the monkey patch below to your code (as per this GitHub issue).

FastText can synthesize a guess-vector, from word-fragments, for any string.

It can work fairly well for typo or variant word-form of a word that was well-represented in training.

For your 'word', 'get up', it might not work so well. There may have been no, or no-meaningful, character-n-grams in the training set of substrings of your 'word' like 'get ', 'et u', or 't up'. But as FastText uses a collision- and presence- oblivious hash-table for storing the n-gram vectors, these will still return essentially-random vectors.

If you want instead something based on the per-word vectors for 'get' and 'up', I think you'd want to use the .get_sentence_vector() method, instead:

https://github.com/facebookresearch/fastText/blob/master/python/README.md#model-object

I do not think this is a fair assessment of the FastText model. It was trained on much longer sentences than you are using for your quick test, so is a sort of train-test data mismatch. I would also guess that most of the Chinese data that the model used at the training time were not in Latin script and there it might have problems with it.

There exist other models for language identification:

• langid.py uses simple trigram statistics.

• langdetect is a port of an old open-source project by Google that uses a simple ML model over character statistics.

• Spacy has a language detection extension.

• Polyglot toolkit for multilingual NLP also has language detection.

However, I would suspect that all of them will have problems with such short text snippets. If this is really how your data look like, then the best thing would be training your own FastText model with the training data matching your use case. For instance, if you are only interested in detecting Chinese, you can classify into two classes: Chinese and non-Chinese.

Emscripten provide a USE_ES6_IMPORT_META flag! Maybe this can solve your problem. Take a look at https://github.com/emscripten-core/emscripten/blob/master/src/settings.js. There is a simple explanation about this flag.

UPDATE

Use USE_ES6_IMPORT_META=0