TFIDF | simple TFIDF | Math library

Below lines need correction:

A vector size of 1000 dimensions is very uncommon, and would require massive amounts of data to train. For example, the famous GoogleNews vectors were for 3 million words, trained on something like 100 billion corpus words - and still only 300 dimensions. Your STEMMED_TOKENS may not be enough data to justify 100-dimensional vectors, much less 300 or 1000.

A choice of min_count=1 is a bad idea. This algorithm can't learn anything valuable from words that only appear a few times. Typically people get better results by discarding rare words entirely, as the default min_count=5 will do. (If you have a lot of data, you're likely to increase this value to discard even more words.)

Are you examining the model's size or word-to-word results at all to ensure it's doing what you expect? Despite your colum being named STEMMED_TOKENS, I don't see any actual splitting-into-tokens, and the Word2Vec class expects each text to be a list-of-strings, not a string.

Finally, without seeing all your other choices for feeding word-vector-enriched data to your other classification steps, it is possible (likely even) that there are other errors there.

Given that a binary-classification model can always get at least 50% accuracy by simply classifying every example with whichever class is more common, any accuracy result less than 50% should immediately cause suspicions of major problems in your process like:

• misalignment of examples & labels
• insufficient/unrepresentative training data
• some steps not running at all due to data-prep or invocation errors

The main problem is the way you are returning the numeric values. x.number.values will return an array of shape (n_samples,) which the FeatureUnion object will try to combine with the result of the transformation of the text features later on. In your case, the dimension of the transformed text features is (n_samples, 98) which cannot be combined with the vector you get for the numeric features.

An easy fix would be to reshape the vector into a 2d array with dimensions (n_samples, 1) like the following:

As stated here, nested parameters must be accessed by the __ (double underscore) syntax. Depending on the depth of the parameter you want to access, this applies recursively. The parameter use_idf is under:

features > text_features > tfidf > use_idf

So the resulting parameter in your grid needs to be:

There is nothing wrong with your code by itself. But your step-by-step approach is using bad practice in Machine Learning theory:

Do not resample your testing data

In your step-by-step approach, you resample all of the data first and then split them into train and test sets. This will lead to an overestimation of model performance because you have altered the original distribution of classes in your test set and it is not representative of the original problem anymore.

What you should do instead is to leave the testing data in its original distribution in order to get a valid approximation of how your model will perform on the original data, which is representing the situation in production. Therefore, your approach with the pipeline is the way to go.

As a side note: you could think about shifting the whole data preparation (vectorization and resampling) out of your fitting and testing loop as you probably want to compare the model performance against the same data anyway. Then you would only have to run these steps once and your code executes faster.

Imagine you trained an AI to recognize a plane using pictures of the motors, wheels, wings and of the pilot's bowtie. Now you're calling this same AI and you ask it to predict the model of a plane with the bowtie alone. That's what scikit-learn is telling you : there are much less features (= columns) in X_unseen than in X_train or X_test.

It's the Combiner's fault. You are specifying in the driver class that you want to use MR2Reducer both as a Combiner and a Reducer in the following commands:

In Wikipedia you can see how to calculate Tf-idf

N - number of documents in corpus.

So similarity depends on number of all documents/sentences in corpus.

If you have more documents/sentences then it changes results.

If you add the same document/sentence few times then it also changes results.