isomap | jeremy stober contact | Data Manipulation library

I need example on how to use isomap from sklearn for dimensionalty reduction of high-dimensional space defined in numpy array.

I am doing some exercises with MNIST digits data but it fails when I try to visualize it. The exercise is from a book BTW. So I import the the dataset

I am drawing four network graphs of the same data but using different properties to color the node. Currently, I am able to generate all 4 networks and for each network I am able to double click a node to see the connected nodes of the network. My question is how do I extend this functionality so that if I click a node on any of the graph networks, the other 3 also show me the node I clicked and the connected nodes therein.

This question asked previously Mouseover event on two charts at the same time d3.js does a similar thing for pie charts but I am not sure how I can adapt it to my code.

I have a square matrix D (currently represented as a numpy array of shape (572, 572)) plausibly corresponding to pairwise distances between points along the surface of a roughly cylindrical object. I.e., the value D[i,j] corresponds to the minimal length of any path along the surface of that hollow cylinder. How do I construct a 3-dimensional (or n-dimensional) embedding of those 572 points into euclidean space which preserves those geodesic distances?

Algorithms like locally linear embedding and isomap are able to take that matrix of pairwise geodesic distances and output an embedding so that the pairwise euclidean distances are the same as the original geodesics. While this is not the same task in general, in the case where the output happens to approximate a hypercube in some dimension the desired transformation has actually happened (consider the swiss roll) since the embedding is itself a manifold, so euclidean distance corresponds to geodesic distance.

This is not the case for even slightly more complicated objects like cylinders. By treating geodesic distances as euclidean, antipodal points on the desired cylinder are mapped to locations much further from each other than desired, and the corresponding global optimization problem will often result in a branching structure with ends of the branches corresponding to maximally distant antipodal points, amplifying small perturbations in the random sampling of the cylinder. In general, naive applications of these algorithms doesn't seem to solve the problem at hand.

Another somewhat fruitful (though expensive) approach has been a brute monte carlo technique. I generate random samples from tubelike objects with varying parameters till I find a set of parameters generating geodesic distance matrices similar to mine, up to a permutation (which is dealt with not too inefficiently by solving the linear system converting that distance matrix to mine and testing to see if the result is near a permutation matrix). Then a near-optimal mapping from my 572 points onto that object preserving pairwise distances is performed by finding the nearest permutation matrix to the aforementioned near-permutation matrix.

This is yielding plausible results, but it presupposes the shape of the data and is horrendously expensive. I've performed some of the more obvious optimizations like working with small random samples instead of the entire data set and using gradient-based techniques for parameter estimation, but a more general-purpose technique would be nice.

This problem of course does not have a unique solution. Even assuming that manifolds can be unambiguously identified in 3-space from a finite uniform sampling, just squishing a cylinder yields a shape with the same geodesics and different euclidean distances (hence a different embedding). This does not bother me any more than LLE and Isomap yielding differing solutions, and I would be fine with any plausible answer.

With regards to uniquely identifying manifolds from a finite sample, for the sake of argument I would be fine just using the dist_matrix_ attribute from a fitted Isomap class from the scikit-learn package without any special parameters to find the geodesics. That performs an unnecessary MDS step, but it isn't terribly expensive, and it works out of the box. We would then like an embedding which minimizes the frobenius distance between the original geodesic distance matrix and the dist_matrix_ attribute.

I have been completing Microsoft's course DAT210X - Programming with Python for Data Science.

When creating SVC models for Machine Learning we are encouraged to split out the dataset X into test and train sets, using train_test_split from sci-kit learn, before performing preprocessing e.g. scaling and dimension reduction e.g. PCA/Isomap. I include a code example, below, of part of a solution i wrote to a given problem using this way of doing things.

However, it appears to be much faster to preprocess and PCA/IsoMap on X before splitting X out into test and train and there was a higher accuracy score.

My questions are:

1) Is there a reason why we can't slice out the label (y) and perform pre-processing and dimension reduction on all of X before splitting out to test and train?

2) There was a higher score with pre-processing and dimension reduction on all of X (minus y) than for splitting X and then performing pre-processing and dimension reduction. Why might this be?

I have a LIST called 'samples', I am loading several images into this LIST from 2 different folders, let's say Folder1 and Folder2. Then I convert this list to a DataFrame and plot them in a 2D scatter plot. I want the scatter plot to show all contents from Folder1 to be Red color and all contents from Folder2 to be in blue color. How can I accomplish this. My code is below:

I created the following graph with help of two functions written by Vincent Zoonekynd (you can find them here) (find my code at the end of the post).

In order to be able to explain what that neighbourhood graph and that parameter "k" is, which the Isometric Feature Mapping uses. "k" specifies how many points each point is directly connected to. Their distance is just the euclidian distance to each other. The distance between any point and its (k + 1)-nearest point (or any point farther away) is called "geodesic", and is the smallest sum of all the lengths of the edges needed to get there. This is sometimes much longer than the euclidian distance. This is the case for points A and B in my figure.

Now I want to add a black line showing the geodesic distance from point A to point B. I know about the command segments(), which will probably be the best for adding the line, and I know that one algorithm to find the shortest path (geodesic distance) is Dijkstra's Algorithm and that it is implemented in the package igraph. However, I'm neither able to have igraph interpret my graph nor to find out the points (vertices) that need to be passed (and their coordinates) on my own.

By the way, if k = 18, i.e. if every point is directly connected to the 18 nearest points, the geodesic distance between A and B will just be the euclidian distance.