K-Means-Clustering | creates clusters using K-Means Clustering Algorithm | Machine Learning library

I'm reasonably confident this works as intended, but please correct me if you spot an error. (cobbled together from geeks for geeks):

We can use the distance measure that is used in the kmodes algorithm to assign each new row to its nearest cluster.

Iris dataset contains 4 features describing the three different types of flowers (i.e. 3 classes). Therefore, each point in the dataset is located in a 4-dimensional space and the same applies to the centroids, so to describe their position you need the 4 coordinates.

In examples, it's easier to use 2-dimensional data (sometimes 3-dimensional) as it is easier to plot it out and display for teaching purposes, but the centroids will have as many coordinates as your data has dimensions (i.e. features), so with the Iris dataset, you would expect the 4 coordinates.

With clustering, there's no meaningful order or comparison between clusters, we're just finding groups of observations that have something in common. There's no reason to refer to one cluster as 'the blue cluster' vs 'the red cluster' (unless you have some extra knowledge about the domain). For that reason, sklearn will arbitrarily assign numbers to each cluster.

The problem may be with the format of your data. Most models will expect a data frame

Your linked guide is, with its given data, a bit of misguided. You can't get meaningful 100-dimensional word-vectors (the gensim Word2Vec class default) from a mere 30-word corpus. The results from such a model will be nonsense, useless for clustering or other downstream steps – so any tutorial purporting to show this process, with true results, should be using far more data.

If you are in fact using far more data, and have succeeded in clustering words, the Word2Vec model's most_similar() function will give you the top-N (default 10) nearest-words for any given input word. (Specifically, they will be returned as (word, cosine_similarity) tuples, ranked by highest cosine_similarity.)

The Word2Vec model is of course oblivious to the results of clustering, so you would have to filter those results to discard words outside the cluster of interest.

I'll assume that you have some lookup object cluster, that for cluster[word] gives you the cluster ID for a specific word. (This might be a dict, or something that does a KMeans-model predict() on the supplied vector, whatever.) And, that total_words is the total number of words in your model. (For example: total_words = len(w2v_model.wv). Then your logic should be roughly like

You could try calculating the adjusted Rand index between two results. This gives a score between -1 and 1, where 1 is a perfect match.

Or by taking argmax of confusion matrix:

scale() returns a matrix, and dplyr/tibble isn't automatically coercing it to a vector. By changing your mutate_all() call to the below, we can have it return a vector. I identified this is what was happening by calling class(df1$speed_scaled) and seeing the result of "matrix".

1. Choose the algorithm wisely. There are clever algorithms, and there are stupid algorithms for kmeans. Lloyd's is stupid, but the only one you will find in GPUs so far. It wastes a lot of resources with unnecessary computations. Because GPU and "big data" people do not care about resource efficiency... Good algorithms include Elkan's, Hamerly's, Ying-Yang, Exponion, Annulus, etc. - these are much faster than Lloyd's.

   Sklearn is one of the better tools here, because it at least includes Elkan's algorithm. But if I am not mistaken, it may be making a dense copy of your data repeatedly. Maybe in chunks so you don't notice it. When I compared k-means from sklearn with my own spherical k-means in Python, my implementation was many times faster. I can only explain this with me using sparse optimizations while the sklearn version performed dense operations. But maybe this has been improved since.

2. Implementation quality is important. There was an interesting paper about benchmarking k-means. Let me Google it:

   Kriegel, H. P., Schubert, E., & Zimek, A. (2017). The (black) art of runtime evaluation: Are we comparing algorithms or implementations?. Knowledge and Information Systems, 52(2), 341-378.

   They show how supposedly the same algorithm can have orders f magnitude runtime differences, depending on implementation differences. Spark does not fare very well there... It has too high overheads, too slow algorithms.

3. You don't need all the data.

   K-means works with averages. The quality of the mean very slowly improves as you add more data. So there is little use in using all the data you have. Just use a large enough sample, and the results should be of almost the same quality. You can exploit this also for seeding. Run on a smaller set first, then add more data for refinement.

4. Because your data is sparse, there is a high chance that k-means is not the right tools anyway. Have you tested the quality of your results? How do you ensure attributes to be appropriately scaled? How much is the result determined simply by where the vectors are 0, and not by the actual non-zero values? Do results actually improve with rerunning k-means so often? What if you d not rerun k-means ever again? What if you just run it on a sample as discussed in 3)? What if you just pick k random centers and do 0 iterations of k-means? What is your best Silhouette? Chances are that you cannot measure the difference and are just wasting time and resources for nothing! So what do you do to ensure reliability of your results?