Text_Analytics | A space for code and projects around analysing news content | Analytics library

thanks for the direct question. I am here after I received your email. I think the two questions that you raised are somewhat similar and can be answered through each other. Basically, your question is why do we need word count vector while conducting sentiment analysis.

In all honesty, this is actually a long answer but I will try to make it as concise as possible. I am not aware of your level of NLP understanding at the moment but all machine learning models are only built for numerical values which means when you are working with text data, you first need to convert the text into a numerical format. This process is known as vectorization. That is essentially what we are doing here but there are many ways of achieving that. The vectorizer that is being used here is a CountVectorizer where each word in the counts dictionary is treated as a separate feature for that particular sentence. This leads to the creation of a sparse matrix which can represent m sentences with n unique words as a m x n matrix.

The way we're going about it is that we count the number of times a word occurs a particular type of sentence (either positive or negative). It is understandable that words like terrible might have a very high count in negative sentences and almost 0 counts in positive sentences. Similarly, there will be a reverse effect for words like 'great' and 'amazing'. This is what is used in classifiers to allot weights to each word. Negative weights to words occurring popularly in negative classes and positive weights to words occurring in positive classes. This is what sentiment analysis classification is based on.

This might be a really helpful resource. You can also read through this.

PS: I wouldn't recommend using TuriCreate before you have either coded this from scratch to understand how it works or used scikit-learn because TuriCreate abstracts a lot of the usage and you might not understand what is happening in the background.