matrix.sh | Send messages via matrix chat protocol right | Chat library

Given a data frame "df," I am trying to print a correlation matrix to display the upper triangle so that it does not display the duplicate correlation coefficients. I want to output the correlation coefficients only where the correlation is +/- 0.7 or greater.

Command:

I'm following the answer to this question and this scikit-learn tutorial to remove artifacts from an EEG signal. They seem simple enough, and I'm surely missing something obvious here.

The components extracted don't have the same length as my signal. I have 88 channels of several hours of recordings, so the shape of my signal matrix is (88, 8088516). Yet the output of ICA is (88, 88). In addition to being so short, each component seems to capture very large, noisy-looking deflections (so out of 88 components only a couple actually look like signal, the rest look like noise). I also would have expected only a few components to look noisy. I suspect I'm doing something wrong here?

The matrix of (channels x samples) has shape (88, 8088516).

Sample code (just using a random matrix for minimum working purposes):

I'm having a hard time vectorializing the following function:

Just learning some rust. I'm using ndarray and I need to construct a zero matrix copying the dimension from another matrix. I tried

Given a matrix m and a pair of "counts" count_x and count_y I would like a new larger matrix that has every value in m repeated a different number of times. So, for example, the m[i,j] block in the new array would have size (count_y[i],count_x[j]).

Here is what I have figured out by looping:

I have a Matrix class, and another class which is using that matrix changing it a little bit. I'd like to test both matrix, one from the matrix class and the other one which has been changed, so I can confirmed they're not the same.

Something like this.

I am sorry if this is a long post, but i have some questions related to Confusion Matrix metric and Cross-Validation that i really need help with.

This picture from Sklearn CV link, shows that our whole dataset should be split into train and test. Then, the train set is split again into a validation part and we train our model in k-1 folds and validate in the remaining one (repeat this k times). And lastly, we test our model with the test set from the beggining.

In my problem, i have a dataset for a unbalanced binary classification problem with 42372 samples. 3615 belong to class 1, the rest are class 0.

Since my dataset is unbalanced, i was using StratifiedShuffleSplit with 5 folds, and got this:

As result, using a MLPClassfier i got the following confusion matrix:

As you can see from that matrix, half my dataset is being used for test (19361+19+1782+28 = 21190).

After this, i changed the CV strategy, and tried StratifiedKfold:

And, as Confusion Matrix, i got this:

As you can see from this second confusion matrix, my whole dataset is being used for test (38644+113+3329+286 = 42372).

So, here are my questions:

1 - Do i need to split my whole data into train/test (e.g., using train_test_split), and then feed CV iterators (KFold, StratifiedKFold, StratifiedShuffleSplit, etc) only with the train part? Or should i feed my whole data into the iterators and they will do the job of splitting it into train/test and split again this train into train and validation?

2 - About the CV strategies i tried, why StratifiedShuffleSplit is using half the data? and why StratifiedKFold uses all the data? Any of those CV is wrong? Are both wrong or are both correct? What i am missing here?

EDIT: The original code to generate the Confusion Matrix i found here. I have just modified it a little bit to fit my needs, and here it goes:

When I run the following with warning flags I get a type conversion warning.

When I use pandas.DataFrame.corr() to create a correlation matrix, I found the correlation matrix(corr_matrix) has 37 columns and the DataFrame(all_data) has 80 columns. In my mind, these two columns should be the same. In another word, the correlation matrix should have the shape (80 x 80). But this did not happen. I have imputed all missing data before creating the correlation matrix. So why the two columns are not equal?

The code