tf-idf | A rubygem that calculates the tf-idf of a corpus

I am currently working with python NLTK to preprocess text data for Kaggle SMS Spam Classification Dataset. I have completed the following steps during preprocessing:

1. Removed any extra spaces
2. Removed punctuation and special characters
3. Converted the text to lower case
4. Replaced abbreviations such as lol,brb etc with their meaning or full form.
5. Removed stop words
6. Tokenized the data

Now I plan to perform lemmatization and stemming separately on the tokenized data followed by TF-IDF done separately on lemmatized data and stemmed data.

Questions are as follows:

• Is there a practical use case to perform lemmatization on the tokenized data and then stem that lemmatized data or vice versa
• Does the idea of stemming the lemmatized data or vice versa make any sense theoretically, or is it completely incorrect.

Context: I am relatively new to NLP and hence I am trying to understand as much as I can about these concepts. The main idea behind this question is to understand whether lemmatization or stemming together make any sense theoretically/practically or whether these should be done separately.

Questions Referenced:

• Should I perform both lemmatization and stemming?: The answer to this question was inconclusive and not accepted, it never discussed why you should or should not do it in the first place.
• What is the difference between lemmatization vs stemming?: Provides the ideas behind stemming and lemmatization but I was unable to conclude the answers to my questions based on this
• Stemmers vs Lemmatizers: Explains the pros and cons, as well as the context in which stemming and lemmatization, might help
• NLP Stemming and Lemmatization using Regular expression tokenization: The question discusses the different preprocessing steps and does stemming and lemmatization separately

So I'm doing tf-idf for very large corpus(100k documents) and it is giving me memory errors. Is there any implantation that can work well with such large number of documents? I want to make my own stopwords list. Also, it worked on 50k documents, what is the limit of number of documents I can use in this calculation if there is one (sklearn implantation).

I'm experimenting with the text analysis tools in sklearn, namely the LDA topic extraction algorithm seen here.

I've tried feeding it other data sets and in some cases I think I would get better topic extraction results if the vector representation of the tf-idf 'features' could allow for phrases.

As an easy example:

I often get top word associations like:

• income
• net
• asset
• fixed
• wealth
• fiscal

Which is understandable, but I think that I won't get the granularity I need for a useful topic extraction unless the TfidfVectorizer() or some other parameter can be tweaked such that I get phrases. Ideally, I want:

• fixed income
• asset management
• wealth management
• net income
• fiscal income

To make things simple, I'm imagining I supply the algorithm with a white list of tolerable 2-word phrases. It would count only those phrases as unique while applying normal tf-idf weighting to all other word entries throughout the corpus.

The documentation for TfidfVectorizer() doesn't seem to support this, but I'd imagine this is a fairly common need in practice -- so how do practitioners go about it?

I am trying code from this page. I ran up to the part LR (tf-idf) and got the similar results

After that I decided to try GridSearchCV. My questions below:

1)

I have a fake news detection problem and it predicts the binary labels "1"&"0" by vectorizing the 'tweet' column, I use three different models for detection but I want to use the ensemble method to increase the accuracy but they use different vectorezer.

I have 3 KNN models the first and the second one vectorizes the 'tweet' column using TF-IDF.

Have been working in Google Sheets on a general table containing approximately a thousand texts. In one column derived form the column containing the texts in their original "written" form, are ngrams (words and the like) extracted from them, and listed in alphabetic order, one list of ngrams corresponding to each text. I’ve been trying without success to derive a second column, from these lists of such ngrams, from which I want to remove instances of certain ngrams of which I have a list (a long list, hundreds of ngrams, and a list to which I could make additions later). In other words, from the text mining vocabulary, I want to remove stop words from lists of tokens.

I tried with SPLIT and REGEXREPLACE functions, or a combination of both, but with no success.

I'm having two dataframes in pandas, the one is initial one:

and the other is the result of TF-IDF operation. So basically, Name was grouped by Group and then sklearn TF-IDF was applied like that:

I am a begginer in NLP and I am using TF-IDF method to apply then a ML model. If I have a dataset like this

1. For the below test text,