textclean | Tools for cleaning and normalizing text data | Regex library

I am importing an Excel file into R, where the date format in Excel is "27-02-2012". However, once I import the dataset into R with the code below:

I'm developing simple text classification for SMS and the full model will be 3 steps:

1. TextCleaning() "Custom function"
2. TfidfVectorizer() "Vectorizer"
3. MultinomialNB() "Classification model"

I wanted to merge the 3 steps in one model using sklearn.pipeline and save the model using joblib.dump, The problem is when load the saved model the output is fixed every time with any test or training data of spam class I got ham!

This is the custom function before Pipeline :

I have downloaded the street abbreviations from USPS. Here is the data:

I have to do a topic modeling based on pieces of texts containing emojis with R. Using the replace_emoji() and replace_emoticon functions let me analyze them, but there is a problem with the results.

A red heart emoji is translated as "red heart ufef". These words are then treated separately during the analysis and compromise the results.

Terms like "heart" can have a very different meaning as can be seen with "red heart ufef" and "broken heart" The function replace_emoji_identifier() doesn't help either, as the identifiers make an analysis hard.

Dummy data set reproducible with by using dput() (including the step force to lowercase:

I am trying to conduct tokenization of n-grams (between 1 (minimum) and 3(maximum)) on my data. After applying this function , I can see that it strips some relevant words such as [sad](words that I have converted from emojis).

For example the input is:

• I dislike lemons [sad]

When I apply the n-gram tokenizer and assess their frequency (which are separated by "_") the output for sad appears like this (bare in mind that I am only printing the top 100 n-grams and other words are included but I want to assess this one specifically):

• [_sad]
• [_sad _]

How do I make sure that "[" its not stripped during tokenization of n-grams? (i.e. In order to become [sad])

This is my code and I am using quanteda package:

I've built a spell check function for a sample of 1000 rows to ensure its efficiency, using the 'hunspell' package and the Australian English dictionary. The spell-checker ignores abbreviations. My actual data has close to 2 million lines, I therefore need to convert the 'for' loops into the 'apply' family functions.

I'm almost there I, but the last part isn't working. Below are the original for loop functions:

I am working in R and using the replace_emoticon function from the textclean package to replace emoticons with their corresponding words:

I am trying to clean my text data and replace Emojis with words so that I can perform a sentiment analysis later on.

Therefore, I am using the replace_emoji function from the textclean package. This should replace all emojis with their corresponding words.

The dataset I am working with is a text corpus, that is also the reason why I used the VCorpus function from the tm package in my sample code below:

I'm doing feature extraction from labelled Twitter data to use for predicting fake tweets. I've been spending a lot of time on various GitHub methods, R libraries, stackoverflow posts, but somehow I couldn't find a "direct" method of extracting features related to emojis, e.g. number of emojis, whether the tweet contains emoji(1/0) or even occurrence of specific emojis(that might occur more often in fake/real news). I'm not sure whether there is a point in showing reproducible code.

"Ore" library, for example, offers functions that gather all tweets in an object and extracts emojis, but the formats are problematic (at least, to me) when trying to create features out of the extractions, as mentioned above. The example below uses a whatsapp text sample. I will add twitter data from kaggle to make it somewhat reproducible. Twitter Dataset: https://github.com/sherylWM/Fake-News-Detection-using-Twitter/blob/master/FinalDataSet.csv