fastText | Library for fast text representation and classification | Natural Language Processing library

Incase someone has similar question, I'll paste the reply I got when asking this question in the Gensim Disussion Group for reference:

You can try it, but I wouldn't expect it to work well for most purposes.

The build_vocab() call establishes the known vocabulary of the model, & caches some stats about the corpus.

If you then supply another corpus – & especially one with more words – then:

• You'll want your train() parameters to reflect the actual size of your training corpus. You'll want to provide a true total_examples and total_words count that are accurate for the training-corpus.
• Every word in the training corpus that's not in the know vocabulary is ignored completely, as if it wasn't even there. So you might as well filter your corpus down to just the words-of-interest first, then use that same filtered corpus for both steps. Will the example texts still make sense? Will that be enough data to train meaningful, generalizable word-vectors for just the words-of-interest, alongside other words-of-interest, without the full texts? (You could look at your pref-filtered corpus to get a sense of that.) I'm not sure - it could depend on how severely trimming to just the words-of-interest changed the corpus. In particular, to train high-dimensional dense vectors – as with vector_size=300 – you need a lot of varied data. Such pre-trimming might thin the corpus so much as to make the word-vectors for the words-of-interest far less useful.

You could certainly try it both ways – pre-filtered to just your words-of-interest, or with the full original corpus – and see which works better on downstream evaluations.

More generally, if the concern is training time with the full corpus, there are likely other ways to get an adequate model in an acceptable amount of time.

If using corpus_file mode, you can increase workers to equal the local CPU core count for a nearly-linear speedup from number of cores. (In traditional corpus_iterable mode, max throughput is usually somewhere in the 6-12 workers threads, as long as you ahve that many cores.)

min_count=1 is usually a bad idea for these algorithms: they tend to train faster, in less memory, leaving better vectors for the remaining words when you discard the lowest-frequency words, as the default min_count=5 does. (It's possible FastText can eke a little bit of benefit out of lower-frequency words via their contribution to character-n-gram-training, but I'd only ever lower the default min_count if I could confirm it was actually improving relevant results.

If your corpus is so large that training time is a concern, often a more-aggressive (smaller) sample parameter value not only speeds training (by dropping many redundant high-frequency words), but ofthen improves final word-vector quality for downstream purposes as well (by letting the rarer words have relatively more influence on the model in the absense of the downsampled words).

And again if the corpus is so large that training time is a concern, than epochs=100 is likely overkill. I believe the GoogleNews vectors were trained using only 3 passes – over a gigantic corpus. A sufficiently large & varied corpus, with plenty of examples of all words all throughout, could potentially train in 1 pass – because each word-vector can then get more total training-updates than many epochs with a small corpus. (In general larger epochs values are more often used when the corpus is thin, to eke out something – not on a corpus so large you're considering non-standard shortcuts to speed the steps.)

-- Gordon

You can use your model name with the Model.get_model_path() method to retrieve the path of the model file or files on the local file system. If you register a folder or a collection of files, this API returns the path of the directory that contains those files.

More info you may want to refer: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-advanced-entry-script#azureml_model_dir

Please check if given references help to work around:

As per the error the arg0 (1st positional argument) should be a str(string). fasttext.load_model accepts first argument as string or char and second argument is utf-8 encoding which is optional.

See if fasttext.load_model(str) can only load files from the local filesystem. Try to copy the data to the local filesystem and then load it from there, e.g. Check this reference from Stack Overflow. Try to download blob to a file and load that file

Unfortunately, the np.fromfile() method on which this load depends doesn't work on a streamed-from-S3 file.

Some alternate options include:

• download the S3 file to a local path first, then use load_facebook_vectors() from there; or…
• while having the FastText file local, load it locally, then use Python's pickle functionality to save it to a single file (now of Python's format), then put that file on S3, and in the future re-load it using Python's unpickling

The utility functions in gensim.utils pickle() and unpickle() (which take a file path, including S3 URLs) may be helpful for the 2nd option, eg:

https://radimrehurek.com/gensim/utils.html#gensim.utils.unpickle

Since your prior code only shows using the vectors (via .load_facebook_vector), not the whole model, you could just pickle & upload the model.wv subcomponent of the loaded model, rather than the whole model, to save some storage/bandwidth.

If perhaps in future Gensim versions, the FastText-model related classes change in shape/operation, an old pickled-model might not cleanly load. In such an eventuality, you could potentially either:

• go back to the original Facebook-format model file (which could then be loaded, & then re-saved in a modern format, again); OR...
• load your pickled model into the older Gensim where it works, save it locally using Gensim's native .save() (which may split it over multiple local files), then in the newer Gensim use Gensim's native FastText.load() to load those older files (which will usually handle older formats), then re-pickle that loaded model, for future re-unpickles into the matching latest Gensim.

There's no one 'best' set of word-vectors: it always depends on your data & downstream goals.

The 'loss' that's optimized, & reported, during FastText training is for the model's internal word-to-nearby-word goal. It is only a guide, via its overall trend & eventual inability to improve further, as to whether more of that kind of training can improve on that internal goal. It is not the case that a model that can reach a lower loss has better metaparameters, or is better at any real downstream tasks.

So: if reported loss was still noticeably decreasing from epoch to epoch when training stopped, it may be worth trying a longer run of more iterations, with all the other data/parameters the same, that instead reaches a point of no-further improvement ('convergence' of the underlying optimization). But don't use FastText-training loss to choose between models with different other metaparameters.

For that, you should use some other repeatable quantititave evaluation of the final word-vectors, ideally in a task as close as possible to your real usage. That is: really plug alternate versions of them into your next step, and review how well they work, & how different sets influence where the full system works better, or worse.

This might be very manual & ad hoc at first: running a set of familiar challenges, and merely 'eyeballing' whethere one 'seems to' be giving more desirable answers or not. But to do well, & truly search all the possibilities for data preprocessing & model metaparameters, you'd ideally want to use some large, automated, & potentially growing-over-time set of probes you can score as 'better' or 'worse'.

The automated tests used in original word-vector papers are often based on some task like analogy-solving, or matching a human's native language reports of which words should be 'closer' to each other than another. Sometimes it makes sense to try to re-use those, as interim evaluations, but ultimately what makes a word-vector perform best on those may not always be what works in other tasks. (In particular, I've seen where word-vectors worse at analogies work noticeably better as inputs to a classifier.)

I doubt FastText is the right approach for this.

Unlike in natural-languages, where word roots/prefixes/suffixes (character n-grams) can be hints to meaning, most invoice number schemes are just incrementing numbers.

Every '###' or '####' is going to have a similar frequency. (Well, perhaps there'd be a little bit of a bias towards lower digits to the left, for Benford's Law-like reasons.) Unless the exact same invoice numbers repeat often* throughout the corpus, so that the whole token, & its fragments, acquire a word-like meaning from surrounding other tokens, FastText's post-training nearest-neighbors are unlikely to offer any hints about correct numbers. (For it to have a chance to help, you'd want the same invoice-numbers to not just repeat many times, but for a lot of those appeearances to have similar OCR errors - but I strongly suspet your corpus instead has invoice numbers only on individual texts.)

Is the real goal to correct the invoice-numbers, or just to have them be less-noisy in a model thaat's trained on a lot more meaningful, text-like tokens? (If the latter, it might be better just to discard anything that looks like an invoice number – with or without OCR glitches – or is similarly so rare it's likely an OCR scanno.)

That said, statistical & edit-distance methods could potentially help if the real need is correcting OCR errors - just not semantic-context-dependent methods like FastText. You might get useful ideas from Peter Norvig's classic writeup on "How to Write a Spelling Corrector".

I solved this issue for a couple of days and each time I thought it was worth reading the documentation on python.net . As a result, I found a solution and it turned out to be quite banal, it is necessary to represent X_vec not as a float[] , but as a List

Per the NotImplementedError, those are the one kind of full Facebook FastText model, -supervised mode, that Gensim does not support.

So sadly, the answer to "How do you load these?" is "you don't".

The .vec files contain just the full-word vectors in a plain-text format – no subword info for synthesizing OOV vectors, or supervised-classification output features. Those can be loaded into a KeyedVectors model:

If you train your own word-vector model, then it will contain vectors for all the words you told it to learn. If a word that was in your training data doesn't appear to have a vector, it likely did not appear the required min_count number of times. (These models tend to improve if you discard rare words who few example usages may not be suitably-informative, so the default min_words=5 is a good idea.)

It's often reasonable for downstream tasks, like feature engineering using the text & set of word-vectors, to simply ignore words with no vector. That is, if some_rare_word in model.wv is False, just don't try to use that word – & its missing vector – for anything. So you don't necessarily need to find, or train, a set of word-vectors with every word you need. Just elide, rather than worry-about, the rare missing words.

Separate observations:

• Stemming/lemmatization & stop-word removal aren't always worth the trouble, with all corpora/algorithms/goals. (And, stemming/lemmatization may wind up creating pseudowords that limit the model's interpretability & easy application to any texts that don't go through identical preprocessing.) So if those are required parts of laerning exercise, sure, get some experience using them. But don't assume they're necessarily helping, or worth the extra time/complexity, unless you verify that rigrously.
• FastText models will also be able to supply synthetic vectors for words that aren't known to the model, based on substrings. These are often pretty weak, but may better than nothing - especially when they give vectors for typos, or rare infelcted forms, similar to morphologically-related known words. (Since this deduced similarity, from many similarly-written tokens, provides some of the same value as stemming/lemmatization via a different path that required the original variations to all be present during initial training, you'd especially want to pay attention to whether FastText & stemming/lemmatization mix well for your goals.) Beware, though: for very-short unknown words – for which the model learned no reusable substring vectors – FastText may still return an error or all-zeros vector.
• FastText has a supervised classification mode, but it's not supported by Gensim. If you want to experiment with that, you'd need to use the Facebook FastText implementation. (You could still use a traditional, non-supervised FastText word vector model as a contributor of features for other possible representations.)