Word2Vector | Self complemented word embedding methods | Icon library

I think what you are trying to achieve is the following:

1. Obtain vector representations from word2vec for every word in your sentence.
2. Average all word vectors of a sentence to obtain a sentence representation.
3. Compute cosine similarity between the vectors of two sentences.

While the code for 2 and 3 looks fine to me in general (haven't tested it though), the issue is probably in step 1. What you are doing in your code with

word2vec_model=gensim.models.Word2Vec(sentences, size=100, min_count=5)

is to initialize a new word2vec model. If you would then call word2vec_model.train(), gensim would train a new model on your sentences so you can use the resulting vectors for each word afterwards. But, in order to obtain useful word vectors that capture things like similarity, you usually need to train the word2vec model on a lot of data - the model provided by Google was trained on 100 billion words.

What you probably want to do instead is to use a pretrained word2vec model and use it with gensim in your code. According to the documentation of gensim, this can be done with the KeyedVectors.load_word2vec_format method.

Your code snippet will be thread-safe if the following conditions are met:

1. The word2vectors.wordsNearest(...) call is thread-safe
2. The word2vectors data structure is created and initialized by the current thread.
3. Nothing changes the data structure between the pools.execute call and the computation finishing.

If wordsNearest doesn't look at other data structures, and if it doesn't change the word2vectors data structure, then it is a reasonable assumption that it is thread-safe. However, the only way to be sure is to analyse it.

But it is also worth noting that you are only submitting a single task to the executor. Since each task is work to be run by a single thread, your code effectively uses only one thread. To exploit multi-threading you need to split your single big task into multiple small and independent tasks; e.g. put the 10,000 repetitions loop outside of the execute call ...

The initial training used info about known words to plot them in a useful N-dimensional space.

It is of course theoretically possible to then use new information, about new words, to also give them coordinates in the same space. You would want lots of varied examples of the new words being used together with the old words.

Whether you want to freeze the positions of old words, or let them also drift into new positions based on the new examples, could be an important choice to make. If you've already trained a pre-existing classifier (like a sentiment classifier) using the older words, and didn't want to re-train that classifier, you'd probably want to lock the old words in place, and force the new words into compatible positioning (even if the newer combined text examples would otherwise change the relative positions of older words).

Since after an effective train-up of the new words, they should generally be near similar-meaning older words, it would be reasonable to expect classifiers that worked on the old words to still do something useful on the new words. But how well that'd work would depend on lots of things, including how well the original word-set covered all the generalizable 'neighborhoods' of meaning. (If the new words bring in shades of meaning of which there were no examples in the old words, that area of the coordinate-space may be impoverished, and the classifier may have never had a good set of distinguishing examples, so performance could lag.)

It depends on how your corpuses and test examples are structured and pre-processed.

You are probably using the pre-trained word-vectors to turn text into numerical features. At first, text examples are vectorized to train the classifier. Later, other (test/production) text examples will be vectorized in the same, and presented to get the classifier to get its judgements.

So you will need to use the same text-to-vectors process for test/production text examples as was used during training. Perhaps you've done that in a separate earlier bulk step, in which case you already have the features in the vector form the classifier uses. But often your classifier pipeline will itself take raw text, and vectorize it – in which case it will need the same pre-trained (word)->(vector) mappings available at test time as were available during training.