twitter-sentiment-analysis | performs Sentiment Analysis on Twitter | Predictive Analytics library

if you have strings in column then you would have to split every sentence into list of words and then put all list in one list - you can use it sum() for this - it should give you all words. To get unique words you can convert it to set() - and later you can convert back to list()

But at start you would have to clean sentences to remove chars like ., ?, etc. I uses regex to keep only some chars and space. Eventually you would have to convert all words into lower or upper case.

When doing machine learning, we want to learn an algorithms that performs well on new (unseen) data. This is called generalization.

The purpose of the test set is, amongst others, to verify the generalization behavior of your classifier. If your model predicts the same labels for each test instance, than we cannot confirm that hypothesis. The test set should be representative of the conditions in which you apply it later.

As a rule of thumb, I like to think that you keep 50-25% of their data as a test set. This of course depends on the situation. 30/4000 is less than one percent.

A second point that comes to mind is that when your classifier is biased towards one class, make sure each class is represented nearly equally in the training and validation set. This prevents the classifier from 'just' learning the distribution of the whole set, instead of learning which features are relevant.

As a final note, normally we report metrics such as precision, recall and F_β=1 to evaluate our classifier. The code in your sample seems to report something based on the global sentiment in all tweets, are you sure that is what you want? Are the tweets a representative collection?

Your dataset is highly imbalanced. You yourself mentioned it in one of the comment, you have 550 positive and 550 negative labelled tweets but 4000 neutral that's why it always favours the majority class. You should have equal number of utterances for all classes if possible. You also need to learn about evaluation metrics, then you'll see most probably your recall is not good. An ideal model should stand good on all evaluation metrics. To avoid overfitting some people also add a fourth 'others' class as well but for now you can skip that.

Here's something you can do to improve performance of your model, either (add more data) oversample the minority classes by adding possible similar utterances or undersample the majority class or use a combination of both. You can read about oversampling, undersampling online.

In this new datset try to have utterances of all classes in this ratio 1:1:1 if possible. Finally try other algos as well with hyperparameters tuned through grid search,random search or tpot.

edit: in your case irrelevant is the 'others' class so you now have 4 classes try to have dataset in this ratio 1:1:1:1 for each class.

One should preprocess the text before feeding into the model, following is the minimal working script(adapted from https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis/):

The example you are trying to run is desinged for running in spark 1.5. You should either download spark 1.5 or run the spark-submit from spark 2.1.0 but with kafka package related to 2.1.0, for example: ./bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0.

How good did you expect, and how good did you achieve?

Combining the three datasets may not improve overall sentiment-detection ability, if the signifiers of sentiment vary in those different domains. (Maybe, 'positive' tweets are very different in wording than product-reviews or movie-reviews. Tweets of just a few to a few dozen words are often quite different than reviews of hundreds of words.) Have you tried each separately to ensure the combination is helping?

Is your performance in line with other online reports of using roughly the same pipeline (Doc2Vec + LinearRegression) on roughly the same dataset(s), or wildly different? That will be a clue as to whether you're doing something wrong, or just have too-high expectations.

For example, the doc2vec-IMDB.ipynb notebook bundled with gensim tries to replicate an experiment from the original 'Paragraph Vector' paper, doing sentiment-detection on an IMDB dataset. (I'm not sure if that's the same dataset as you're using.) Are your results in the same general range as that notebook achieves?

Without seeing your code, and details of your corpus-handling & parameter choices, there could be all sorts of things wrong. Many online examples have nonsense choices. But maybe your expectations are just off.

After playing around a bit, I figured out a more elegant solution for this using pandas

Never mind. I got rid of the SelectColumnsTransform altogether (departing from the example experiment), instead using a R script in the training experiment to save the names of the columns selected, then another R script in the predictive experiment to load those names and remove any other feature columns.