gensim | Topic Modelling for Humans | Topic Modeling library

You only use the .wv property to fetch the KeyedVectors object from another more complete algorithmic model, like a full Word2Vec model (which contains a KeyedVectors in its .wv attribute).

If you're already working with just-the-vectors, there's no need to request the word-vectors subcomponent. Whatever you were going to do, you just do to the KeyedVectors directly.

However, you're also using the .vocab attribute, which has been replaced. See the migration FAQ for more details:

https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4#4-vocab-dict-became-key_to_index-for-looking-up-a-keys-integer-index-or-get_vecattr-and-set_vecattr-for-other-per-key-attributes

(Mainly: instead of doing an in w2v_model.wv.vocab, you may only need to do in kv_model or in kv_model.key_to_index.)

By default, to avoid using an unbounded amount of RAM, the Gensim Phrases class uses a default parameter max_vocab_size=40000000, per the source code & docs at:

https://radimrehurek.com/gensim/models/phrases.html#gensim.models.phrases.Phrases

Unfortunately, the mechanism behind this cap is very crude & non-intuitive. Whenever the tally of all known keys in they survey-dict (which includes both unigrams & bigrams) hits this threshold (default 40,000,000), a prune operation is performed that discards all token counts (unigrams & bigrams) at low-frequencies until the total unique-keys is under the threshold. And, it sets the low-frequency floor for future prunes to be at least as high as was necessary for this prune.

For example, the 1st time this is hit, it might need to discard all the 1-count tokens. And due to the typical Zipfian distribution of word-frequencies, that step along might not just get the total count of known tokens slightly under the threshold, but massively under the threshold. And, any subsequent prune will start by eliminated at least everything with fewer than 2 occurrences.

This results in the sawtooth counts you're seeing. When the model can't fit in max_vocab_size, it overshrinks. It may do this many times in the course of processing a very-large corpus. As a result, final counts of lower-frequency words/bigrams can also be serious undercounts - depending somewhat arbitrarily on whether a key's counts survived the various prune-thresholds. (That's also influenced by where in the corpus a token appears. A token that only appears in the corpus after the last prune will still have a precise count, even if it only appears once! Although rare tokens that appeared any number of times could be severely undercounted, if they were always below the cutoff at each prior prune.)

The best solution would be to use a precise count that uses/correlates some spillover storage on-disk, to only prune (if at all) at the very end, ensuring only the truly-least-frequent keys are discarded. Unfortunately, Gensim's never implemented that option.

The next-best, for many cases, could be to use a memory-efficient approximate counting algorithm, that vaguely maintains the right magnitudes of counts for a much-larger number of keys. There's been a litte work in Gensim on this in the past, but not yet integrated with the Phrases functionality.

That leaves you with the only practical workaround in the short term: change the max_vocab_size parameter to be larger.

You could try setting it to math.inf (might risk lower performance due to int-vs-float comparisons) or sys.maxsize – essentially turning off the pruning entirely, to see if your survey can complete without exhausting your RAM. But, you might run out of memory anyway.

You could also try a larger-but-not-essentially-infinite cap – whatever fits in your RAM – so that far less pruning is done. But you'll still see the non-intuitive decreases in total counts, sometimes, if in fact the threshold is ever enforced. Per the docs, a very rough (perhaps outdated) estimate is that the default max_vocab_size=40000000 consumes about 3.6GB at peak saturation. So if you've got a 64GB machine, you could possibly try a max_vocab_size thats 10-14x larger than the default, etc.

Incase someone has similar question, I'll paste the reply I got when asking this question in the Gensim Disussion Group for reference:

You can try it, but I wouldn't expect it to work well for most purposes.

The build_vocab() call establishes the known vocabulary of the model, & caches some stats about the corpus.

If you then supply another corpus – & especially one with more words – then:

• You'll want your train() parameters to reflect the actual size of your training corpus. You'll want to provide a true total_examples and total_words count that are accurate for the training-corpus.
• Every word in the training corpus that's not in the know vocabulary is ignored completely, as if it wasn't even there. So you might as well filter your corpus down to just the words-of-interest first, then use that same filtered corpus for both steps. Will the example texts still make sense? Will that be enough data to train meaningful, generalizable word-vectors for just the words-of-interest, alongside other words-of-interest, without the full texts? (You could look at your pref-filtered corpus to get a sense of that.) I'm not sure - it could depend on how severely trimming to just the words-of-interest changed the corpus. In particular, to train high-dimensional dense vectors – as with vector_size=300 – you need a lot of varied data. Such pre-trimming might thin the corpus so much as to make the word-vectors for the words-of-interest far less useful.

You could certainly try it both ways – pre-filtered to just your words-of-interest, or with the full original corpus – and see which works better on downstream evaluations.

More generally, if the concern is training time with the full corpus, there are likely other ways to get an adequate model in an acceptable amount of time.

If using corpus_file mode, you can increase workers to equal the local CPU core count for a nearly-linear speedup from number of cores. (In traditional corpus_iterable mode, max throughput is usually somewhere in the 6-12 workers threads, as long as you ahve that many cores.)

min_count=1 is usually a bad idea for these algorithms: they tend to train faster, in less memory, leaving better vectors for the remaining words when you discard the lowest-frequency words, as the default min_count=5 does. (It's possible FastText can eke a little bit of benefit out of lower-frequency words via their contribution to character-n-gram-training, but I'd only ever lower the default min_count if I could confirm it was actually improving relevant results.

If your corpus is so large that training time is a concern, often a more-aggressive (smaller) sample parameter value not only speeds training (by dropping many redundant high-frequency words), but ofthen improves final word-vector quality for downstream purposes as well (by letting the rarer words have relatively more influence on the model in the absense of the downsampled words).

And again if the corpus is so large that training time is a concern, than epochs=100 is likely overkill. I believe the GoogleNews vectors were trained using only 3 passes – over a gigantic corpus. A sufficiently large & varied corpus, with plenty of examples of all words all throughout, could potentially train in 1 pass – because each word-vector can then get more total training-updates than many epochs with a small corpus. (In general larger epochs values are more often used when the corpus is thin, to eke out something – not on a corpus so large you're considering non-standard shortcuts to speed the steps.)

-- Gordon

Import GensimWord2VecVectorizer in your Flask Web app python file.

Word2Vec expects a list of lists as input, where the corpus (main list) is composed of individual documents. The individual documents are composed of individual words (tokens). Word2Vec iterates over all documents and all tokens. In your example you have passed a single list to Word2Vec, therefore Word2Vec interprets each word as an individual document and iterates over each word character which is interpreted as a token. Therefore you have built a vocabulary of characters not words. To build a vocabulary of words you can pass a nested list to Word2Vec as in the example below.

The 'current working directory' that the Python process will consider active, and thus will use as the expected location for your plain relative filename GoogleNews-vectors-negative300.bin, will depend on how you launched Flask.

You could print out the directory to be sure – see some ways at How do you properly determine the current script directory? – but I suspect it may just be the /Users/Ile-Maurice/Desktop/Flask/flaskapp/ directory.

If so, you could relatively-reference your file with the path relative to the above directory...

Convert list or that partular format into an numpy array and save it as a .npy file easy to save and easy to read, using this by numpy gives you advantage of loading it in almost every platform like google colab, replit ..... refer to this link for more details on saving a npy file numpy.save()

Using pickle is also a good option but things get a bit tricky at points when difference in encoding standards and such problems arise.

In plotly-python, I don't think there's an easy way of retrieving the location of the cursor. You can attempt to use go.FigureWidget to highlight a trace as described in this answer, but i think you're going to be limited with with plotly-python and i'm not sure if highlighting the closest n points will be possible.

However, I believe that you can accomplish what you want in plotly-dash since callbacks are supported - meaning you would be able to retrieve location of your cursor and then calculate the n closest data points to your cursor and highlight the data points as needed.

Below is an example of such a solution. If you haven't seen it before, it looks complicated, but what is happening is that I am taking the point where you clicked as an input. plotly is plotly.js under the hood so it comes us in the form of a dictionary (and not some kind of plotly-python object). Then I calculate the closest three data points to the clicked input point by comparing the coordinates of every other point in the dataframe, add the information from the three closest points as traces to the input with the color teal (or any color of your choosing), and send this modified input back as the output, and update the figure.

I am using click instead of hover because hover would cause the highlighted points to flicker too much as you drag your mouse through the points.

Also the dash app doesn't work perfectly as I believe there is some issue when you double click on points (you can see me click once in the gif below before getting it to start working), but this basic framework is hopefully close enough to what you want. Cheers!

If a .save() is creating any files with the word trainables in it, you're using a older version fo Gensim. Any new training should definitely prefer using a current version. As of now (January 2022), that's gensim-4.1.2, released 2021-09.

If an attempt at a .load() generated that particular error, then there should've been that file, alongside the others you mention, created when the .save() had been done. (In fact, the only way that the main file you named with path_filename should be able to know that other filename is if that other file was written successfully, allowing the main file to complete writing.)

Are you sure that file wasn't written, but then somehow left behind, perhaps getting deleted or not moving alongside the other few files to some new filesystem path?

In general, I would suggest:

• using latest Gensim for any new training
• always enable Python logging at the INFO level, & watch the logging/console output of training/saving processes closely to see confirmation of expected activity/steps
• keep all files from a .save() that begin with the same main filename (in your examples above, w2v_US.model) together - & keep in mind that for larger models it may be a larger roster of files than for a small test model

You will probably have to re-train the model, but you might be able to re-generate a compatible lockf file via steps like the following:

• save aside all files of any potential use
• from the exact same configuration as your original .save() – including the same outdated Gensim version, exact same model parameters, & exact same training corpus – repeat all the model-building steps you did before up through the .build_vocab() step. (That is: no extra need to .train().) This will create an untrained dummy model that should exactly match the vocabulary 'shape' of your broken model.
• use .save() to save that dummy model again - watching the logs/output for errors. There should be, alongside the other files, a file with a name like dummy.model.trainables.vectors_lockf.npy. If so, you might be able to copy that away, rename it to tbe the file expected by the original model whose load failed, then leave it alongside that original model - and the .load() might then succeed, or fail in a different way.

(If there were other problems/corruption at the time of the original model creation, this might not work. In particular, I wonder if when you talk about retraining the model, you didn't start with a fresh Word2Vec instance, but somehow expanded the older one, which might've added other problems/complications. In that case, a full retraining, ideally in the latest Gensim, would be necessary, and also a better basis for going forward.)