TF-IDF | Term Frequency - Inverse Document Frequency in Ruby | Audio Utils library

As we know Facebook's FastText is a great open-source, free, lightweight library which can be used for text classification. But here a problem is the pipeline seem to be end-to end black-box. Yes, we can change the hyper-parameters from these options for setting training configuration. But I couldn't manage to find a way to access to the vector embedding it generates internally.

Actually I want to do some manipulation on the vector embedding - like introducing tf-idf weighting apart from these word2vec representations and another thing I want to to is oversampling using SMOTE which requires numerical representation. For these reasons I need to introduce my custom code in between the overall pipeline which seems to be inaccessible for me. How introduce custom steps in this pipeline?

I have a dataframe x that is:

I'm trying to get the tf-idf for a set of documents using the following code:

I am trying to create a tf-idf table using TfidfVectorizer from sklearn package in python. For example I have a corpus of one string "PD-L1 expression positive (≥1%–49%) and negative for actionable molecular markers"

TfidfVectorizer has an token_pattern argument that indicates how the token should be like. The default is token_pattern = token_pattern='(?u)\b\w\w+\b', it will split all the words by space and remove the numbers and special characters to create the tokens, and generates some tokens like below

["pd", "expression", "positive","and" ,"negative" ,"for" ,"actionable" ,"molecular" ",markers"]

But something I would like to have is:

["pd-l1", "expression", "positive", "≥1%–49%","and" ,"negative" ,"for" ,"actionable" "molecular" ,"markers"]

I was tweaking token_pattern argument for hours but cannot get it right. Alternatively, Is there here a way to tell explicitly to the vectorizer that I want to havepd-l1 and >1%-49% as token without going too wild on regrex? Any help is very appreciated!

I have a dataset with different type of variables: binary, categorical, numerical, textual.

Usually any search engine software creates inverted indexes to make searches faster. The basic format is:-

word: , , .....

Whenever there is a search query inside quote like "Harry Potter Movies" it means there should be exact match of positions of word and in searches like within k word queries like hello /4 world it generally means that find the word world in the range of 4 word distance either in left or right from the word hello. My question is that we can employ solution like linearly checking the postings and calculating distances of words like in query, but if collection is really large we can't really search all the postings. So is there any other data structure or kind of optimisation lucene or solr uses?

One first solution can be only searching some k postings for each word. Other solution can be only searching top docs(usually called champion list sorted by tf-idf or similar during indexing), but more better docs can be ignored. Both solutions have some disadvantage, they both don't ensure quality. But in Solr server we get assured quality of results even in large collections. How?

I created a text classifier that uses Tf-Idf using sklearn, and I want to use BERT and Elmo embedding instead of Tf-Idf.

How would one do that ?

I'm getting Bert embedding using the code below:

I do a string matching using TF-IDF and Cosine Similarity and it's working good for finding the similarity between strings in a list of strings.

Now, I want to do the matching between a new string against the previously calculated matrix. I calculate the TF-IDF score using below code.

tfidf = TfidfVectorizer(lowercase=False, ) tfidf.fit_transform(questions)

word2tfidf = dict(zip(tfidf.get_feature_names(), tfidf.idf_))