kmeans | k-means-visualization | Data Visualization library

To select a column, remove the () from df(['LTVCluster']):

In python3.7 with yellowbrick v1.2 and scikit-learn v0.23.2 I don't get the error but when I bump scikit-learn to v0.24.0 I get the same ImportError.

I checked the API documentation and, like in the link you referenced, in sklearn v0.22.0 they deprecated utils.safe_indexing to private (utils._safe_indexing). But I don't get the import error with yellowbrick v1.2 and v0.22.0<=sklearn<=v0.23.2 installed. I think they just finally removed the public utils.safe_indexing in v0.24.0 so if you just install a version of scikit-learn before v0.24.0 the import should work.

Edit: Here is yellowbrick's github issue if you want to track their progress on a workaround or update for this problem

For kmeans you can find the inertia_ of it. Which can give you an idea how well kmeans algorithm has worked.

You compute the clusters in four dimensions. Note this implies the centroids are four-dimensional points too. Then you plot two-dimensional projections of the clusters. So when you plot the centroids, you have to pick out the same two dimensions that you just used for the scatterplot of the individual points.

Is something wrong with the procedure I follow?

Yes.

Using TSNE projects data onto another space, on which you have no real control.
Doing so is supposed to keep close points close, and far points far.

You then use KNN on the projected space to determine the groups.
This part loses any grouping information you previously had [citation needed, need to see what the data was beforehand]!

It would make much more sense to color the groups according to some prior labeled data, not according to KNN
-OR-
to use KNN on the original space for grouping, and then to color the projected space according to that grouping.

What you did in fact is meaningless, as it loses all prior information - labels and spatial.

To conclude:

1. If you have labels, use them.
2. If you don't, use a more sophisticated clustering algorithm, starting with KNN on the original space, as you can see KNN on the projected space is not enough.

While assigning ranks, Make sure to group it on the basis of month.

Complete code:

KMeans of sklearn often expect features to be a 2-d array, instead of a 1-d vector as you passed. So you need to modify your X to be an array. Besides, if you want to rely on group-by-combine mechanism, why not put column indexing within the to-apply function, since assigning from such an operation is cumbersome.

I found my mistake. I had to increase the parameter max_points_per_centroid. As I have so many data points it sampled a sub-batch for the fit. For a larger number of clusters this sub-batch is larger. See FAQ of faiss:

max_points_per_centroid * k: there are too many points, making k-means unnecessarily slow. Then the training set is sampled

The larger subbatch of course has a larger inertia as there are more points in total.

I think that your approach is not the best one. Your goal is to select genes associated with censored/uncensored survival. The use of supervised methods seems the most suitable. Using a k-means will only cluster genes by similarities without regard to survival, and even if you wanted to add survival in your modeling it would not make sense because you are omitting censoring.

There are Cox regressions to which an L1 penalty is added, allowing variable selection without omitting censoring. This kind of approach seems more appropriate to accomplish your goal and fits better in your context. To learn more, here is an article from Jiang Gui & Hongzhe Li that uses penalized Cox regression (look at the R package biospear too if needed): https://academic.oup.com/bioinformatics/article/21/13/3001/196819