scaler | CPVPS sets resource requirements on a target resource

Two things: first, you need to pass the transformers or the estimators themselves to the pipeline, not the result of fitting/transforming them (that would give the resultant arrays to the pipeline not the transformers, and it'd fail). Pipeline itself will be fitting/transforming. Second, since you have specific transformations to the specific columns, ColumnTransformer is needed.

Putting these together:

You cannot use the export_text function on the whole pipeline as it only accepts Decision Tree objects, i.e. DecisionTreeClassifier or DecisionTreeRegressor. Only pass the fitted estimator of your pipeline and it will work:

you can do this by passing passthrough to your param_grid like so:

The main problem is the way you are returning the numeric values. x.number.values will return an array of shape (n_samples,) which the FeatureUnion object will try to combine with the result of the transformation of the text features later on. In your case, the dimension of the transformed text features is (n_samples, 98) which cannot be combined with the vector you get for the numeric features.

An easy fix would be to reshape the vector into a 2d array with dimensions (n_samples, 1) like the following:

MinMaxScaler is an object that can fit itself to certain data and also transform that data. There are

• The fit method fits the scaler's parameters to that data. It then returns the MinMaxScaler object
• The transforms method transforms data based on the scaler's fitted parameters. It then returns the transformed data.
• The fit_transform method first fits the scaler to that data, then transforms it and returns the transformed version of the data.

In your example, you are treating the MinMaxScaler object itself as the data! (see 1st bullet point)

The same MinMaxScaler shouldn't be fitted twice on different dataset since its internal values will be changed. You should never fit a minmaxscaler on the test dataset since that's a way of leaking test data into your model. What you should be doing is fit_transform() on the training data and transform() on the test data.

The answer here may also help this explanation: fit-transform on training data and transform on test data

When you call StandardScaler.fit(X_train), what it does is calculate the mean and variance from the values in X_train. Then calling .transform() will transform all of the features by subtracting the mean and dividing by the variance. For convenience, these two function calls can be done in one step using fit_transform().

The reason you want to fit the scaler using only the training data is because you don't want to bias your model with information from the test data.

If you fit() to your test data, you'd compute a new mean and variance for each feature. In theory these values may be very similar if your test and train sets have the same distribution, but in practice this is typically not the case.

Instead, you want to only transform the test data by using the parameters computed on the training data.

As stated here, nested parameters must be accessed by the __ (double underscore) syntax. Depending on the depth of the parameter you want to access, this applies recursively. The parameter use_idf is under:

features > text_features > tfidf > use_idf

So the resulting parameter in your grid needs to be:

The issue is you are training the scaler on a machine with an older verion of sklearn than the machine you're using to load the scaler.

Noitce the UserWarning

UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.23.2 when using version 0.24.0. This might lead to breaking code or invalid results. Use at your own risk. UserWarning)

The solution is to fix the version mismatch. Either by upgrading one sklearn to 0.24.0 or downgrading to 0.23.2

Your output from model.predict_proba() is a matrix with 2 columns, one for each class. To calculate roc, you need to provide the probability of the positive class:

Using an example dataset:

GridSearchCV will help you find the best set of hyperparameter according to your pipeline and dataset. In order to do that it will use cross validation (split the your train dataset into 5 equal subset in you case). This means that your best_estimator will be trained on 80% of the train set.

As you know the more data a model see, the better its result is. Therefore once you have the optimal hyperparameters, it is wise to retrain the best estimator on all your training set and assess its performance with the test set.

You can retrain the best estimator using the whole train set by specifying the parameter refit=True of the Gridsearch and then score your model on the best_estimator as follows: