ngrams | Generate n-grams

Input data (for demo purpose, all strings have been cleaned):

A common error in text transformers of sklearn involves the shape of the data: unlike most other sklearn preprocessors, text transformers generally expect a one-dimensional input, and python's duck-typing causes weird errors from both arrays and strings being iterables.

Your TextTransformer.transform returns X[['Tweet']], which is 2-dimensional, and will cause problems with the subsequent CountVectorizer. (Converting to a numpy array with .values doesn't change the dimensionality problem, but there's also no compelling reason to do that conversion.) Returning X['Tweet'] instead should cure that problem.

We can use combn

You can add a length parameter to your function; then just extend your slices from 1 character to that length:

tf.data.Dataset.map applies a function to each element (a Tensor) of a dataset. The __call__ method of the TextVectorization object expects a Tensor, not a tf.data.Dataset object. Whenever you want to apply a function to the elements of a tf.data.Dataset, you should use map.

You can split the data at each word, use rle to find consecutive occurrence and paste the first value together.

An option is to use a recursive function to convert the values to character from factor (the integer coercion values suggest that the nested list elements are factor class), by default, the how = 'unlist' in rapply), then we wrap those vector with list to create a single list element

Refitting the TF-IDF in order to calculate the score of a single entry is not the way; you should simply use the .transform() method of the existing fitted vectorizer to your new string (not to the whole matrix):

Since you want to combine counts from similar stemmed trigrams you can use a dictionary with frozensets as keys: the keys will be the stemmed trigrams and the values will be the total count.

You have to use frozensets instead sets as keys since the keys of dict must be hashable (which is not the case for the sets).

You will have something like this: