node2vec | The Distributed Node2Vec Algorithm for Very Large Graphs

The error was due to torch.ops.torch_cluster.random_walk returning a tuple instead of an array/tensor. I fixed it by using replacing the functions pos_sample and neg_sample in the torch_geometric.nn.Node2Vec with these.

You can use a seaborn palette to generate 12 different RGB color values and then create a column called color in your dataframe based on the weight values:

Each call to the Word2Vec() constructor creates an all-new model.

However, runs are not completely deterministic under normal conditions, for a variety of reasons, so results quality for downstream evaluations (like your unshown clustering) will jitter from run-to-run.

If the variance in repeated runs with the same data is very large, there are probably other problems, such an oversized model prone to overfitting. (Stability from run-to-run can be one indicator that your process is sufficiently specified that the data and model choices are driving results, not the randomness used by the algorithm.)

If this explanation isn't satisfying, try adding more info to your question - such as the actual magnitude of your evaluation scores, in repeated runs, both with and without the changes that you conjecture are affecting results. (I suspect the variations from the steps you think are having effect will be no larger than variations from re-runs or different seed values.)

(More generally, Word2Vec is generally hungry for as much varies training data as possible; only if texts are non-representative of the relevant domain are they likely to result in a worse model. So I generally wouldn't expect being choosier about which subset of sentences is best to be an important technique, unless some of the sentences are total junk/noise, but of course there's always a change you'll find some effects in your particular data/goals.)

You could consider the shared k-step neighbors as in the Katz index described in this paper: 1

The idea is, roughly speaking, to consider the number of common neighbors, common 2-step neighbors, 3-step neighbors, etc. with some weight decreasing with the step. So direct shared neighbors should count more than shared 3-step neighbors. To save on computation, you could consider only up to 2-step neighbors. Another way to think about this is from a random walk perspective, also discussed in the paper.

I believe most applications of word2vec to graphs give each node a unique ID, which is then used as the 'word' token fed to the algorithm. If your nodes have other values, that repeat, those values aren't ideal as the node-IDs.

(While word2vec doesn't natively handle continuous-magnitudes, there has been some research extending it that way – for example, I think Facebook's 'StarSpace' allows mixing scalar features with the discrete tokens of traditional word2vec. I suppose you could also consider banding ranges of your nodes' scalar dimensions into discrete tokens, which could sometimes be used instead of IDs, to learn embeddings for what a range-of-values might be related to.)

You are probably running out of memory. Watch a readout of the Python process size during your attempts, and optimize your walks iterable to not compose a large in-memory list.

Assuming you've used a standard word2vec library to train your models, each run bootstraps a wholly-separate model whose coordinates are not necessarily comparable to any other model.

(Due to some inherent randomness in the algorithm, or in the multi-threaded handling of training input, even running two training sessions on the exact same data will result in different models. They should each be about as useful for downstream applications, but individual tokens could be in arbitrarily-different positions.)

That said, you could try to synthesize some measures of how much two models are different. For example, you might:

• Pick a bunch of random (or domain-significant) word-pairs. Check the similarity between each pair, in each model individually, then compare those values between models. (That is, compare model1.similarity(token_a, token_b) with model2.similarity(token_a, token_b).) Consider the difference-between-the-models as as some weighted combination of all the tested similarity-differences.

• For some significant set of relevant tokens, collect the top-N most-similar tokens in each model. Compare this lists via some sort of rank-correlation measure, to see how much one model has changed the 'neighborhoods' of each token.

For each of these, I'd suggest verifying their operation against a baseline case of the exact-same training data that's been shuffled and/or trained with a different starting random seed. Do they show such models as being "nearly equivalent"? If not, you'd need to adjust the training parameters or synthetic measure until it does have the expected result - that models from the same data are judged as alike, even though tokens have very different coordinates.

Another option might be to train one giant combined model from a synthetic corpus where:

• all the original unmodified 'texts' from both eras all appear once
• texts from each separate era appear again, but with some random-proportion of their tokens modified with an era-specific modifier. (For example, 'foo' sometimes becomes 'foo_1' when in first-era texts, and sometimes becomes 'foo_2' in second-era texts. (You don't want to convert all tokens in any one text to era-specific tokens, because only tokens that co-appear with each other influence each other, and you thus want tokens from either era to sometimes appear with common/shared variants, but also often appear with era-specific variants.)

At the end, the original token 'foo' will get three vectors: 'foo', 'foo_1', and 'foo_2'. They should all be quite similar, but the era-specific variants will be relatively more-influenced by the era-specific contexts. Thus the differences between those three (and relative movement in the now common coordinate space) will be an indication of the magnitude and kinds of changes that happened between the two eras' data.

you cannot pass a string ('P1.bpmn') into the Node2Vec constructor. It accepts a networkx graph. You should create a networkx graph first, and only then use the Node2Vec constructor

I figured this out. The weight values (stored in unnormalized probabilities) are being added to get a value called 'norm_const', which is then dividing the unnormalized probs. So since they're being added, possibility of zero happening arises, hence zero division error.