gensim-word2vec | 利用gensim训练词向量

Yes, you can think of the index (position) of gensim's Word2Vec word-vectors as being the one dimension that would be 1.0 – with all other V dimensions, where V is the count of unique words, being 0.0.

The implementation doesn't actually ever create one-hot vectors, as a sparse or explicit representation. It's just using the word's index as a look-up for its dense vector – following in the path of the word2vec.c code from Google on which the gensim implementation was originally based.

(The term 'doctags' is only relevant in the Doc2Vec – aka 'Paragraph Vector' – implementation. There it is the name for the distinct tokens/ints that are used for looking up document-vectors, using a different namespace from in-document words. That is, in Doc2Vec you could use 'doc_007' as a doc-vector name, aka a 'doctag', and even if the string-token 'doc_007' also appears as a word inside documents, the doc-vector referenced by doctag-key 'doc_007' and the word-vector referenced by word-key 'doc_007' wouldn't be the same internal vector.)

The gensim Word2Vec model does not expect strings as its text examples (sentences), but lists-of-tokens. Thus, it's up to your code to tokenize your text, before passing it to Word2Vec.

Your code as shown just passes raw data from 'aggregate.txt' file into Word2Vec as wFileRead.

Look at examples in the gensim documentation, including the LineSentence class included with gensim, for ideas

There's no simple formula for the best size - it will depend on your data and purposes.

The best practice is to devise a robust, automatable way to score a set of word-vectors for your purposes – likely with some hand-constructed representative subset of the kinds of judgments, and preferred results, you need. Then, try many values of size (and other parameters) until you find the value(s) that score highest for your purposes.

In the domain of natural language modeling, where vocabularies are at least in the tens-of-thousands of unique words but possibly in the hundreds-of-thousands or millions, typical size values are usually in the 100-1000 range, but very often in the 200-400 range. So you might start a search of alternate values around there, if your task/vocabulary is similar.

But if your data or vocabulary is small, you may need to try smaller values. (Word2Vec really needs large, diverse training data to work best, though.)

Regarding your code-as-shown:

• there's unlikely any point to computing a new model for every item in your dataset (discarding the previous model on each loop iteration). If you want a count of the unique tokens in any one tokenized item, you could use idiomatic Python like len(set(word_tokenize(item))). Any Word2Vec model of interest would likely need to be trained on the combined corpus of tokens from all items.

• it's usually the case that min_count=1 makes a model worse than larger values (like the default of min_count=5). Words that only appear once generally can't get good word-vectors, as the algorithm needs multiple subtly-contrasting examples to work its magic. But, trying-and-failing to make useful word-vectors from such singletons tends to take up training-effort and model-state that could be more helpful for other words with adequate examples – so retaining those rare words even makes other word-vectors worse. (It is most definitely not the case that "retaining every raw word makes the model better", though it is almost always the case that "more real diverse data makes the model better".)

Generally, your approach should work.

It's likely the specific problem you're encountering was caused by an extra probing step you took and is not shown in your code, because you had no reason to think it significant: some sort of most_similar()-like operation on model_w2v_new after its build_vocab() call but before the later, malfunctioning operations.

Traditionally, most_similar() calculations operate on a version of the vectors that has been normalized to unit-length. The 1st time these unit-normed vectors are needed, they're calculated – and then cached inside the model. So, if you then replace the raw vectors with other values, but don't discard those cached values, you'll see results like you're reporting – essentially random, reflecting the randomly-initialized-but-never-trained starting vector values.

If this is what happened, just discarding the cached values should cause the next most_similar() to refresh them properly, and then you should get the results you expect:

I'm not sure if containerization allows containers to share the same memory-mapped files – but even if it does, it's possible that whatever utility you're using to measure per-container memory usage counts the memory twice even if it's shared. What tool are you using to monitor memory usage and are you sure it'd indicate true sharing? (What happens if, outside of gensim, you try using Python's mmap.mmap() to open the same giant file in two containers? Do you see the same, more, or less memory usage than in the gensim case?)

But also: in order to do a most_similar(), the KeyedVectors will create a second array of word-vectors, normalized to unit-length, in property vectors_norm. (This is done once, when first needed.) This normed array isn't saved, because it can always be re-calculated. So for your usage, each container will create its own, non-shared, vectors_norm array - undoing any possible memory savings from shared memory-mapped files.

You can work around this by:

• after loading a model but before triggering the automatic normalization, explicitly force it yourself with a special argument to clobber the original raw vectors in-place. Then save this pre-normed version:

Those docs aren't in the right format: each text should be a list-of-string-tokens, not a string.

And, the same min_count threshold will apply to incremental updates: words less frequent that that threshold will be ignored. (Since a min_count higher than 1 is almost always a good idea, a word that appears only once in any update will never be added to the model.)

Incrementally adding words introduces lots of murky issue with unclear proper choices with regard to model quality, balancing the effects of early-vs-late training, management of the alpha learning-rate, and so forth. It won't necessarily improve your model; with the wrong choices it could make it worse, by adjusting some words with your new texts in ways that move them out-of-compatible-alignment with earlier-batch-only words.

So be careful and always check with a repeatable automated quantitative quality check that your changes are helping. (The safest approach is to retrain with old and new texts in one combined corpus, so that all words get trained against one another equally on all data.)