Document-Similarity | Context-based similar documents with gensim | Topic Modeling library

You can do most of that with SpaCY and some regexes.

So, you have to take a look at the SpaCY API documentation.

Basic steps in any NLP pipeline are the following:

1. Language detection (self explanatory, if you're working with some dataset, you know what the language is and you can adapt your pipeline to that). When you know a language you have to download a correct models from SpaCY. The instructions are here. Let's use English for this example. In your command line just type python -m spacy download en and then import it to the preprocessing script like this:

Depending on the content of your raw_docs, this can be the correct behaviour.

Your code returns an empty tf_idf although your query words appear in your original documents and your dictionary. tf_idf is computed by term_frequency * inverse_document_frequency. inverse_document_frequency is computed by log(N/d), where N is your total number of documents and d is the number of documents a specific term occurs in.

My guess is that your query terms ['client', 'is'] occur in each document of yours, resulting in an inverse_document_frequency of 0 and an empty tf_idf list. You can check this behaviour with the documents I took and modified from the tutorial you mentioned: