tokenizer | A simple tokenizer in Ruby for NLP tasks | Natural Language Processing library

The error states that you do not have a variable called sentences in the scope. I believe the tutorial presumes you already have a list of sentences and are tokenizing it.

Have a look at the documentation The first argument can be either a string or list of string or list of list of strings.

You can make the entire class inherit torch.nn.Module like so:

Here is the situation, using a simple file to disentangle the issue from irrelevant specificities like pickle, Tensorflow, and tokenizers:

The SpaCy tokenizer seems to cache each token in a map internally. Consequently, each new token increases the size of that map. Over time, more and more new tokens inevitably occur (although with decreasing speed, following Zipf's law). At some point, after having processed large numbers of texts, the token map will thus outgrow the available memory. With a large amount of available memory, of course this can be delayed for a very long time.

The solution I have chosen is to store the SpaCy model in a TTLCache and to reload it every hour, emptying the token map. This adds some extra computational cost for reloading the SpaCy model from, but that is almost negligible.

You need to define your own method to instantiate the entity ruler:

Application resources will become embedded resources by the time of deployment, so it is wise to start accessing them as if they were, right now. An embedded-resource must be accessed by URL rather than file. See the info. page for embedded resource for how to form the URL.

Thanks for your work, work with getResource. Here is the working code

You can call tokenizer.decode on the output of the tokenizer to get the words from its vocabulary under given indices:

If you want to include # in your search, you should use different analyzer than standard analyzer because # will be removed during analyze phase. You can use whitespace analyzer to analyze your text field. Also for search you can use wildcard pattern:

Query:

The problem is that the file you are working in is named spacy.py, which is interfering with the spacy module. So you should rename your file to something other than "spacy".