ADASYN | Adaptive Synthetic Sampling Approach for Imbalanced Learning | Machine Learning library

You need to have a test BOW function that should reuse the count vectorizer model that was built during the training phase.

Think about using pipeline for reducing the code verbosity.

To fix this, what I did was resampled all but the two major majority classes, and continued to do so via:

One of the problem to use fillna with df.mean() is that if the column contains only nan (or inf before as you replace by nan), then the column is still full of nan after the fillna. One way is to remove the columns that have only nan, because anyway these columns won't be useful for the ML model. To do so, you can use dropna and chain all the methods.

From a programming perspective, an identical question asked in the relevant Github repo back in 2017 was answered negatively:

[Question]

I have a data frame that I want to apply smote to but I wish to only use a subset of the columns. The other columns contain additional data for each sample and I want each new sample to contain the original info as well

[Answer]

There is no way to do that apart of extracting the column in a new matrix and process it with SMOTE. Even if you generate a new samples you have to decide what to put as values there so I don't see how such feature can be added

Answering from a modelling perspective, this is not a good idea and, even if you could find a programming workaround, you should not attempt it - and arguably, this is the reason why the developer of imbalanced-learn above was dismissive even in the thought of adding such a feature in the SMOTE implementation.

Why is that? Well, synthetic oversampling algorithms, like SMOTE, essentially use some variant of a k-nn approach in order to create artificial samples "between" the existing ones. Given this approach, it goes without saying that, in order for these artificial samples to be indeed "between" the real ones (in a k-nn sense), all the existing (numerical) features must be taken into account.

If, by employing some programming alchemy, you manage at the end to produce new SMOTE samples based only on a subset of your features, putting the unused features back in will destroy any notion of proximity and "betweenness" of these artificial samples to the real ones, thus compromising the whole enterprise by inserting a huge bias in your training set.

In short:

• If you think your Month and year are indeed useful features, just include them in SMOTE; you may get some nonsensical artificial samples, but this should not be considered a (big) problem for the purpose here.

• If not, then maybe you should consider removing them altogether from your training.