kdtree | Some mucking around with KD Trees

According to the documentation, pdist "returns a condensed distance matrix". That means it would try to calculate and return a matrix of about 189090^2/2 = 17877514050 entries, causing your computer run out of ram.

If you want to calculate distances between some specific data points, filter them out before using pdist.

If you really want to calculate the entire distance matrix, it's better to calculate distances of a small partition of data points at a time (e.g. 1000), and save the result in the disk.

I think you spend most of your time in your while loop so I will focus on those lines:

Since y contains disjoint ranges and the union of them is also a range, a very fast solution is to first perform a binary search on y and then count the resulting indices and only return the ones that appear at least 10 times. The complexity of this algorithm is O(Nx log Ny) with Nx and Ny the number of items in respectively x and y. This algorithm is nearly optimal (since x needs to be read entirely).

First of all, you need to transform your current y to a Numpy array containing the beginning value of all ranges (in an increasing order) with N as the last value (assuming N is excluded for the ranges of y, or N+1 otherwise). This part can be assumed as free since y can be computed at compile time in your case. Here is an example:

You need to iterate over and check every tuple in set1, you can do that using a set comprehension, and any():

Check the design rationale of boost geometry for a solution to this problem. The methodology boils down to these steps:

1. Declare a class template that extracts position information from a type, e.g.

I have not worked with the Clark-Evans (CE) index before, but having read the information you linked to and studied your code, my interpretation is this:

• The index value for Dataset2 is less than the index value for Dataset1. This correctly reflects the visual difference in clusteredness, that is, the smaller index value is associated with data that is more clustered.
• It is probably not meaningful to say that two CE index values are similar, other than special cases like observing that two CE index values are both smaller than 1 or both greater than 1, or if A < B < C then AB are more similar than AC.
• The p-value and the index value measure different things. The index value measures degree of clusteredness (if less than 1) or regularity (if greater than 1). The p-value (inversely) measures how certain it is that the data are more clustered than would be expected by chance, or more regular than would be expected by chance. The p-value in particular is sensitive to the sample size as well as the distribution of points.
• The use of pi in calculating SE reflects the assumption of Euclidean distances between points (rather than, say, city block distances). That is, the nearest neighbour of a point is the one at the smallest radial distance. The use of pi in calculating SE does not make any assumptions about the shape of the region of interest.
• Particularly for small datasets (like Dataset2) you will want to track down information about the potential impact of boundary effects on the index value or the p-value.

More speculatively, I wonder if it would be useful to use a convex hull to help determine the region of interest rather than do this subjectively.

Each integer index in indices refers to an index value (row number) of locations_a. You can use locations_a.loc[] to convert these indices to their corresponding station names as a numpy array:

point[axis] is not working because point does not support such bracket notation.

There are several solutions:

1. Define Point as a named tuple:

   Instead of defining Point as something like:

If I'm understanding correctly what you're trying to do, then you need to make two changes:

1. Change the type of l from List to List: