skipgram | word embedding based on a shallow feed

Are you sure your walks corpus is what you expect, and what Gensim Word2Vec expects?

For example, is len(walks) equal to 169343? Is len(walks[0]) equal to 80? Is walks[0] a list of 80 string-tokens?

Note also: by default Word2Vec uses a min_count=5 - so any token that appears fewer than 5 times is ignored during training. In most cases, this minimum – or an even higher one! – makes sense, because tokens with only 1, or a few, usage examples in usual natural-language training data can't get good word-vectors (but can, in aggregate, function as dilutive noise that worsens other vectors).

Depending on your graph, one walk from each node might not ensure that node appears at least 5 times in all the walks. So you could try min_count=1.

But it'd probably be better to do 5 walks from every starting point, or enough walks to ensure all nodes appear at least 5 times. 169,343 * 80 * 5 is still only 67,737,200 training words, with a manageable 169,343 count vocabulary. (If there's an issue expanding the whole training set as one list, you could make an iterable that generates the walks as needed, one by one, rather than all up-front.)

Alternatively, something like 5 walks per starting-node, but of only 20 steps each, would keep the corpus about the same size bu guarantee each node appears at least 5 times.

Or even: adaptively keep adding walks until you're sure every node is represented enough times. For example, pick a random node, do a walk, keep a running tally of each node's appearances so far – & keep growing the net total of every node's You could also try an adaptive corpus that keeps adding walks until every node is represented a minimum number of times.

Conceivably, for some remote nodes, that might take quite long to happen upon them, so another refinement might be: do some initial walk or walks, then tally how many visits each node got, & while the least-frequent node is below the target min_count, start another walk from it – guaranteeing it at least one more visit.

This could help oversample less-connected regions, which might be good or bad. Notably, with natural language text, the Word2Vec sample parameter is quite helpful to discard certain overrepresented words, preventing them from monopolizing training time redundantly, ensuring less-frequent words also get good representations. (It's a parameter which can sometimes provide the double-whammy of less training time and better results!) Ensuring your walks spend more time in less-connected areas might provide a similar advantage, especially if your downstream use for the vectors is just as interested in the vectors for the less-visited regions.

You can view the default parameters for the Gensim Word2Vec model, in an unmodified Gensim library, in the Gensim docs. Here's a link to the current version (4.1) docs for the Word2Vec constructor method, showing all default parameter values:

https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec

class gensim.models.word2vec.Word2Vec(sentences=None, corpus_file=None, vector_size=100, alpha=0.025, window=5, min_count=5, max_vocab_size=None, sample=0.001, seed=1, workers=3, min_alpha=0.0001, sg=0, hs=0, negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=, epochs=5, null_word=0, trim_rule=None, sorted_vocab=1, batch_words=10000, compute_loss=False, callbacks=(), comment=None, max_final_vocab=None, shrink_windows=True)

Two of those parameters – hs=0, negative=5 – mean the default mode has hierarchical-softmax disabled, and negative-sampling enabled with 5 negative words. These have been the default of Gensim's Word2Vec for many versions, so even other code is using an older version, this is likely the mode used (unless parameters or modified/overriden code changed them).

This looks like the frequent-word-downsampling feature common in word2vec implementations. (In the original Google word2vec.c code release, and the Python Gensim library, it's adjusted by the sample parameter.)

In practice, it's likely sampling_table has been precalculated so that the rarest words are always used, common words skipped a little, and the very-most-common words skipped a lot.

That seems to be the intent reflected by the comment for make_sample_table().

You could go ahead and call that with a probe value, like say 1000 for a 100-word vocabulary, and see what sampleing_table it gives back. I suspect it'll be numbers close to 1.0 early (drop lots of common words), and close to 0.0 late (keep most/all rare words).

This tends to improve word-vector quality, by reserving more relative attention for medium- and low-frequency words, and not exessively overtraining/overweighting plentiful words.

Good question. Here are two methods, one relying on regular expressions on the corpus text, and the second using (as @Kohei_Watanabe suggests in the comment) using window for tokens_select().

First, create some sample text.

Probably the easiest way is wildcarding to represent the "skip".

Running your code on a clean Python 3.7 conda environment should work after installing fasttext with pip (pip install fasttext).

If you do that, you should see in a Linux console with

Check out this Out-of-memory issues section on their github page.

Often it's because that batch size or sequence length is too large to fit in the GPU memory, followings are the maximum batch configurations for a 12GB memory GPU, as listed in the above link

Let's say you have a corpus of size n a window size of m, so that the total window size considered at a given time is 2m+1. Then the number of skip grams would be,