ensemble | Ensemble : listen to music

Indeed, Regularizations are constraints that are added to the loss function. The model when minimizing the loss function will have to also minimize the regularization term. Hence, This will reduce the model variance as it cannot overfit.

Acceptable parameters for l2_regularization are often on a logarithmic scale between 0 and 0.1, such as 0.1, 0.001, 0.0001.

TL;DR

This problem might occur when y_train is not of the type that is inputted into any classifier model. In this case, y_train was of the type Series. When I changed the type to a NumPy array, it worked fine. Here's the code:

Can you try this for debug?

This is happening because you have a text transformer object in your pipeline. The problem with this approach is that the pipeline will pass the whole dataframe to the TfidfVectorizer. However, the text transformers of scikit-learn expect a 1d input.

Passing a 2d dataframe to TfidfVectorizer causes some weird processing where it mistakes the column names as documents. You can check with this simple example:

There is nothing wrong with your code by itself. But your step-by-step approach is using bad practice in Machine Learning theory:

Do not resample your testing data

In your step-by-step approach, you resample all of the data first and then split them into train and test sets. This will lead to an overestimation of model performance because you have altered the original distribution of classes in your test set and it is not representative of the original problem anymore.

What you should do instead is to leave the testing data in its original distribution in order to get a valid approximation of how your model will perform on the original data, which is representing the situation in production. Therefore, your approach with the pipeline is the way to go.

As a side note: you could think about shifting the whole data preparation (vectorization and resampling) out of your fitting and testing loop as you probably want to compare the model performance against the same data anyway. Then you would only have to run these steps once and your code executes faster.

Try calling the evaluate() like this:

You can average-ensemble them like this:

make_union just unions multiple datasets, and FunctionTransformer(copy) duplicates all the columns. So the nested make_union and FunctionTransformer(copy) makes several copies of each feature. That seems very odd, except that with ExtraTreesClassifier it will have an effect of "bootstrapping" the feature selections. See also Issue 581 for an explanation for why these are generated in the first place; basically, adding copies is useful in stacking ensembles, and the genetic algorithm used by TPOT means it needs to generate those first before exploring such ensembles. There it is recommended that doing more iterations of the genetic algorithm may clean up such artifacts.

After that things are straightforward, I guess: you perform a univariate feature selection, and fit an extra-random trees classifier.