cosine-similarity | Measures similarity between two Strings

I am stucked at getting the correlation between product groups within an order in my dataset in python. I am using a pandas data frame. I want to know if some product group combinations (e.g. shirt with shoes) correlate.

My dataframe looks like this:

As you can see, if the order has more than one product, the order is split into multiple rows.

I've tried to group the order_ids and use pandas corr() function, but I need two inputs for that, and I only have one (product_group).

Maybe I need something like cosine-similarity?

Thanks for helping me out on this! I appreciate any help :)

I have two dataframes. df1['column'] has 70k unique text values. df2['column'] has 20 unique text values.

I want to find the closest synonym for all the 70k values by looking at the 20 values in df2['column']. and want an additional column in df1, which has the best synonym for that word.

I found a code where you could do semantic search and gives the top 5 synonyms with a score.

I'm doing a semantic search to find the closest synonym in two text columns, in two different dataframes.

The code is as below,

I suspect this is something very fundamental I don't know or understand about this code; my only excuse is that I am a complete beginner in python.

I am trying some of the cosine similarity matrix calculations from this post:

What's the fastest way in Python to calculate cosine similarity given sparse matrix data?

One of them requires the calculation of the reciprocal of the diagonal of the initial matrix product.
Say that he initial matrix is m, each row of which represents an 'object', whose 'coordinates' are in the columns of the matrix. So you want to calculate cosine similarities between rows.
Then, to use the matrix product method, you do something like mp = numpy.dot(m, m.T).

Now, if there are no rows with only 0's in m, the diagonal of mp can never have any zero values, as each of its elements is the sum of the squared elements of the corresponding row of m.
The m I am using in my calculations has indeed no rows with all 0's.
And indeed, when I do:

I have been working with some R packages that calculate (cosine) (sparse) similarity matrices from sparse binary matrices, e.g. proxyC.

As I am now starting (and learning) to use python as well, and I was told it might even be faster, I would like to try and run the same calculations there.

I found this interesting post:

What's the fastest way in Python to calculate cosine similarity given sparse matrix data?

which describes a few methods.

I did try some of them out after writing out a small test matrix myself by hand.
Now I would like to try on 'real' data.
And that's where I encounter a problem I currently cannot solve.

My data come in tsv files that associate objects (ID's) to comma-separated lists of features (FP's). E.g.:

I have seen many stack overflow questions about similarity matrix but they deal with RDD or other cases and I could not find the direct answer to my problem and I decided to post a new question.

I have a list of movies and a list of tropes. To calculate the similarity between two movies, I am using cosine differences. If all the weights are even, then it simplifies pretty well:

I have trained my word2vec model from gensim and I am getting the nearest neighbors for some words in the corpus. Here are the similarity scores:

I have two tensors a & b of shape (m,n), and I would like to compute a distance matrix m using some distance metric d. That is, I want m[i][j] = d(a[i], b[j]). This is somewhat like cdist(a,b) but assuming a generic distance function d which is not necessarily a p-norm distance. Is there a generic way to implement this in PyTorch?

And a more specific side question: Is there an efficient way to perform this with the following metric