word2vec-google | word2vec wordembedding embedding google | Topic Modeling library

No: your code won't work for your intents.

When you execute the line...

You're generally doing the right thing, but note:

• the most_similar() method also disqualifies from its results any of the named words provided - so even if 'king' is (still) the closest word to the result, it will be ignored. Your formulation might very well have 'queen' as the next-closest word, after ignoring the input words - which is all that the 'analogy' tests need.

• the most_similar() method also does its vector-arithmetic on versions of the vectors that are normalized to unit length, which can result in slightly different answers. If you change your uses of model.wv['king'] to model.get_vector('king', norm=True), you'll get the unit-normed vectors instead.

See also similar earlier answer: https://stackoverflow.com/a/65065084/130288

If the Jupyter kernel is dying without a clear error message, you are likely running out of memory.

There may be more information logged to the console where you started the Jupyter server. If you expand you question to include any info there, as well as details about the model you've loaded (size on disk) and system you're running on (especially, RAM available), it may be possible to make other suggestions.

Also:

Whereas gensim-3.8.3 requires a big new increment of RAM when the first .most_similar() call is made, the gensim-4.0.0beta pre-release only needs a much-smaller increment at that time - so it is far more likely that if a model succeeds in loading, you should also be able to get .most_similar() results. So it would also be useful to know:

• How did you install the gensim-4.0.0beta, and did you confirm that's the version actually used by your notebook kernel's environment?
• Are you certain that the prior steps (such as loading) have succeeded, and that it's only & exactly the most_similar() that's triggering the failure? (Is it in a separate cell, and before attempting the most_similar() can you query other aspects of the model, such as its length or whether it contains certain words, successfully?)

The problem with the first path is that you are loading fastText embeddings like word2vec embeddings and word2vec can't cope with Out Of Vocabulary words.

The good thing is that fastText can manage OOV words. You can use Facebook original implementation (pip install fasttext) or Gensim implementation.

For example, using Facebook implementation, you can do:

This much vary on various factors (models, cluster resourcing, pipeline) but trying to answers to your main question :

1). Spark pipeline might solve your problem if they fits your needs in terms of the Tokenizers, Words2Vec, etc. However those are not so powerful as the one already available of the shelf and loaded with api.load. You might also want to take a look to Deeplearning4J which brings those to Java/Apache Spark and see how it can do the same things: tokenize, word2vec,etc

2). Following the current approach I would see loading the model in a foreachParition or mapPartition and ensure the model can fit into memory per partition. You can shrink down the partition size to a more affordable number based on the cluster resources to avoid memory problems (it's the same when for example instead of creating a db connection for each row you have one per partition).

Typically Spark udfs are good when you apply a kind o business logic that is spark friendly and not mixin 3rd external parties.

It is supposed to throw an error if you ask for a word that's not present because you chose not to learn vectors for rare words, like 'country' in your example. (And: such words with few examples usually don't get good vectors, and retaining them can worsen the vectors for remaining words, so a min_count as large as you can manage, and perhaps much larger than 1, is usually a good idea.)

The fix is to do one of the following:

1. Don't ask for words that aren't present. Check first, via something like Python's in operator. For example:

You may want to look at the implementation of _save_word2vec_format() in gensim for an example of Python code which writes that format:

https://github.com/RaRe-Technologies/gensim/blob/e859c11f6f57bf3c883a718a9ab7067ac0c2d4cf/gensim/models/utils_any2vec.py#L104

Are you trying to load a pretrained model and run an inference? By initializing are you referring to loading a model or initializing new weights for each instance this is executed?