NER | BiLSTM-ELMo-CNN-CRF for CoNLL | Natural Language Processing library

Make sure you are really using spaCy 3, in case you haven't :)

You can check this from the console by running python -c "import spacy; print(spacy.__version__)"

By issuing via command line pip install spacy==3.0.6 in a python env, and then running in the python console

Not sure if the final format is json, yet below is an example to process the data into the print format, i.e.

The problem is that the file you are working in is named spacy.py, which is interfering with the spacy module. So you should rename your file to something other than "spacy".

You won't be able to train these at the same time, if the dataset is not the same.

If you're working with spaCy v3, it should be relatively straightforward to combine the two training steps into one final pipeline. For instance, create a config that trains the NER first, and store it to disk. Then, create a new config where you source the NER from the previously trained pipeline, and then define this NER component as frozen:

The error is caused by the transformer model that flair loads. The cache directory for transformers has to be specified in additional by setting the environment variable TRANSFORMERS_CACHE=/path/to/transformers

You haven't really provided enough information here, but it looks like you can't hold all the spaCy docs in memory.

A very simple workaround for this would be to split your CSV file up and process it one chunk at a time.

Another thing you can do, since it looks like you're just saving some words, is to avoid saving the docs by changing your for loop a bit.

You are looking for the named entity recognition pipeline (token classification):

The description of your data is a little vague but given these assumptions:

1. You don't know if a document is type A or type B, you need to classify it.
2. The NER is completely different between type A and B documents.

What you should do is use (up to) three separate spaCy pipelines. Use the first pipeline with a textcat model to classify docs into A and B types, and then have one pipeline for NER for type A docs and one pipeline for type B docs. After classification just pass the text to the appropriate NER pipeline.

This is not the most efficient possible pipeline, but it's very easy to set up - you just train three separate models and stick them together with a little glue code.

You could also train the models separately and combine them in one spaCy pipeline, with some kind of special component to make execution of the NER conditional, but that would be pretty tricky to set up so I'd recommend the separate pipelines approach first.

That said, depending on your problem it's possible that you don't need two NER models, and learning entities for both types of docs would be effective. So I would also recommend you try putting all your training data together, training just one NER model, and seeing how it goes. If that works then you can have a single pipeline with textcat and NER models that don't directly interact with each other.

To respond to the comment, when I say "pipeline" I mean a Language object, which is what spacy.load returns. So you train models using the config and each of those is in a directory and then you do this:

I am no longer looking for an answer because I figured it out.

The discussions at the following links:

https://github.com/explosion/spaCy/discussions/8178

spacy 3 NER Scorer() throws TypeError: score() takes 2 positional arguments but 3 were given

were very useful and I was able to write the following code (in case anyone that reads the current discussion may still find it difficult to make the transition):