knnimpute | Python implementations of kNN imputation | Data Visualization library

There is actually one way to check best K where there is no need to split between train & test.

The method is to study the Density with different K numbers, but it is just for One variable (I will select the one with more imputations needed). The one nearest to original distribution is the best to select.

In every iteration, you remove outliers from X_train_transformed and assign the returned values back to X_train_transformed. Your criteria for removing outliers is such that some values will always be removed (see below).

As for whether it is normal behavior of the dataset, Yes!. Any numerical dataset will have a mean and std, and will most probably have values for which (value - mean) / std will be greater than 3. If you remove such values and calculate a new mean and std, you will now have new values for which (value - mean) / std will be greater than 3 since your mean and std will have changed.

I would recommend only removing outliers once. Maybe play around with the threshold to determine how many you want to remove. Also, consider reading up how normal distributions, their means, and standard deviations work.

Below is an example of how you could compare the classifier's accuracy for different parameter combinations using 5-fold cross-validation and visualize the results.

You need to create a pipeline for each column type to make sure that the different steps are applied sequentially (i.e. to make sure that the missing values are imputed prior to encoding and scaling), see also this example in the scikit-learn documentation.

After passing the imputer, the non-imputed columns are moved to the right as noted in notes under the documentation:

Columns of the original feature matrix that are not specified are dropped from the resulting transformed feature matrix, unless specified in the passthrough keyword. Those columns specified with passthrough are added at the right to the output of the transformers.

We can try just using the imputer first:

The sklearn KNNImputer uses the nan_euclidean_distances metric as a default. According to its user guide

If a sample has more than one feature missing, then the neighbors for that sample can be different depending on the particular feature being imputed.

The algorithm might use different sets of neighborhoods to impute the single missing value in column D and the two missing values in column A.

This is a simple implementation of the KNNImputer:

If you just want to coerce numbers to the nearest integer value, you'll want to round the values in the column to the nearest whole number before converting them to integers. Otherwise, your fractional values would all be truncated instead of rounded.

I would suggest setting the value (-999) to np.nan and then use fillna() with method='ffill'. It propagates the last-valid value to the NA values.

Note, if the first element in each column is np.nan it is not filled (since there is no value before it to propagate)

Your KNNImputer has used the parameter add_indicator=True, so the additional columns are presumably missingness indicators for some of your numeric columns.