dtw | Dynamic Time Warping in Python | Computer Vision library

You can use window functions to derive the required sort criteria, assuming you are using MySql 8

You can save the cluster groupings while retrieving them for plotting.

The optimal number of clusters is based either on your presumptions, e.g. equal to the highest number of items, or you can determine it empirically. To do that, you run the algorithm for different numbers of k and calculate the error of the clustering, for example by calculating MSE between all members of a cluster and the cluster center. Then you'd have to make a decision between the acceptable error (which most likely decreases with the number of clusters) and whether a larger number of clusters still makes sense for the task at hand.

To reduce time complexity of the empirical approach, you can change three variables: the maximum for K, the number of iterations and the number of samples used in the parameter sweep. If you want the most optimal solution, you are best served to take the day to run this. However, if you are hard pressed for time or suspect you need to rerun this at some point, I advise you to use a subset of your data for hyperparameter searches.

More practically speaking, I think you will find k << len(items, so your search range can probably be greatly reduced. Combine that with a subset of the data points and you should save a lot of time.

I can suggest reducing the number of features which i think its 20 features from your dataset shape, Which mean you have 20 dimensions.

You can reduce the number of features by using PCA (Principal Component Analysis) like the following:

to_date transforms a string to a date, meaning all the "time" part (hours/minutes/seconds) is lost. You should use to_timestamp function instead of to_date, as follows:

Under IEEE 754 rules, inf + anything is inf. Since you filled your matrix with infinities, then read a value out of it and added another, you can't help but get an infinite result.

I assume the code of both implementations are correct and as been carefully checked (otherwise the benchmark would be pointless).

The issue likely comes from the compilation time of the function. Indeed, the first call is significantly slower than next calls, even with cache=True. This is especially important for the parallel implementation as compiling parallel Numba code is often slower (since it is more complex). The best solution to avoid this is to compile Numba functions ahead of time by providing types to Numba.

Besides this, benchmarking a computation only once is usually considered as a bad practice. Good benchmarks perform multiple iterations and remove the first ones (or consider them separately). Indeed, several other problems can appear when a code is executed for the first time: CPU caches (and the TLB) are cold, the CPU frequency can change during the execution and is likely smaller when the program is just started, page faults may need to be needed, etc.

In practice, I cannot reproduce the issue. Actually, p_calc is 3.3 times faster on my 6-core machine. When the benchmark is done in a loop of 5 iterations, the measured time of the parallel implementation is much smaller: about 13 times (which is actually suspicious for a parallel implementation using 6 threads on a 6-core machine).

If I understand correctly, you want to compute the distances using awarp, but that distance function takes signals of varying length. So you need to avoid creating an array, because NumPy doesn't allow 'ragged' arrays. Then I think you can do this:

It appears the solution in my case was to include the index in the array. For example, if your data looks like this: