nlp | 中文自然语言处理、网页抽取、关键词抽取、命名实体识别等HTTP接口实现

Imagine that your dataframe is

You could use collections.Counter to count the words:

While I do not know how to deal with this problem directly, I had a somewhat similar issue(and solved). The difference is:

• I use fairseq
• I can run my code on google colab with 1 GPU
• Got RuntimeError: unable to mmap 280 bytes from file : Cannot allocate memory (12) immediately when I tried to run it on multiple GPUs.

From the other people's code, I found that he uses python -m torch.distributed.launch -- ... to run fairseq-train, and I added it to my bash script and the RuntimeError is gone and training is going.

So I guess if you can run with 21000 samples, you may use torch.distributed to make whole data into small batches and distribute them to several workers.

Make sure you are really using spaCy 3, in case you haven't :)

You can check this from the console by running python -c "import spacy; print(spacy.__version__)"

By issuing via command line pip install spacy==3.0.6 in a python env, and then running in the python console

Here you go, a few simple steps:

1. Get counts per product and rating

The SpaCy tokenizer seems to cache each token in a map internally. Consequently, each new token increases the size of that map. Over time, more and more new tokens inevitably occur (although with decreasing speed, following Zipf's law). At some point, after having processed large numbers of texts, the token map will thus outgrow the available memory. With a large amount of available memory, of course this can be delayed for a very long time.

The solution I have chosen is to store the SpaCy model in a TTLCache and to reload it every hour, emptying the token map. This adds some extra computational cost for reloading the SpaCy model from, but that is almost negligible.

You can use the head index. E.g.,

You need to define your own method to instantiate the entity ruler:

Here is how I would do it: