NER | Named entity recognition system using multi-stage CRF | Natural Language Processing library

spacy version: 3.1.4 does not have the feature from_glove.

I was able to use nlp.vocab.vectors.from_glove() in spacy version: 2.2.4.

If you want, you can change your spacy version by using:

!pip install spacy==2.2.4 on your Jupyter cell.

I don't know how this code could have worked, since the result of sequenceS has a constraint >>, which cannot be met by an interface due to possible declaration merging (see this TypeScript issue, for example). And that constraint was already present in 2019, which is way before the blog post was written.

Anyhow, you can get the example to work by declaring OrderHKD as a type rather than an interface:

It seems there are two separate problems.

One is that you seem to already have records in the topics, and if you have it configured to earliest the app will read all those records when it starts up. You can either set ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to latest, or, if you're running locally on docker, you can stop the Kafka container and prune the volumes with something like docker system prune --volumes (note that this will erase data from all your stopped containers - use wisely).

Can you try one of these and test again?

The other problem is that the framework is wrongly setting the default maxDelay of 30s even though the annotation states the default is to ignore. I'll open an issue for that and add the link here.

For now You can set a maxDelay such as @Backoff(delay = 600000, multiplier = 3.0, maxDelay = 5400000), then the application should have the correct delays for 10, 30 and 90 minutes as you wanted.

Let me know if that works out for you, or if you have any other problems related to this issue.

EDIT: Issue opened, you can follow the development there https://github.com/spring-projects/spring-kafka/issues/2137

It should be fixed in the next release.

EDIT 2: Actually the phrasing in the @BackOff annotation is rather ambiguous, but seems like the behavior is correct and you should explicitly set a larger maxDelay.

The documentation should clarify this behavior in the next release.

EDIT 3: To answer your question in the comments, the way retryable topics work is the partition is paused for the duration of the delay, but the consumer keeps polling the broker, so longer delays don't trigger rebalancing.

From your logs the rebalancing is from the main topic's partitions, so it's unlikely it has anything to do with this feature.

EDIT 4: The Retryable Topics feature was released in Spring for Apache Kafka 2.7.0, which uses kafka-clients 2.7.0. However, there have been several improvements to the feature, so I recommend using the latest Spring Kafka version (currently 2.8.3) if possible to benefit from those.

Please try this:

Taking the path from your error message:

After you obtained the body with plain text, you can parse the text into a document and get a list of all entities with their labels and texts, and then instantiate a Pandas dataframe with those data:

Most labels have definitions you can access using spacy.explain(label).

For NORP: "Nationalities or religious or political groups"

For more details you would need to look into the annotation guidelines for the resources listed in the model documentation under https://spacy.io/models/.

The vectors setting is not related to the transformer or what you're trying to do.

In the new config, you want to use the source option to load the components from the existing pipeline. You would modify the [component] blocks to contain only the source setting and no other settings:

First, spaCy does no transformation of the input - it takes it literally as-is and preserves the format. So you don't lose any information when you provide text to spaCy.

That said, input to spaCy with the pretrained pipelines will work best if it is in natural sentences with no weird punctuation, like a newspaper article, because that's what spaCy's training data looks like.

To that end, you should remove meaningless white space (like newlines, leading and trailing spaces) or formatting characters (maybe a line of ----?), but that's about all the cleanup you have to do. The spaCy training data won't have bullets, so they might get some weird results, but I would leave them in to start. (Also, bullets are obviously printable characters - maybe you mean non-ASCII?)

I have no idea what you mean by "muck with the indexes", but for some older NLP methods it was common to do more extensive preprocessing, like removing stop words and lowercasing everything. Doing that will make things worse with spaCy because it uses the information you are removing for clues, just like a human reader would.

Note that you can train your own models, in which case they'll learn about the kind of text you show them. In that case you can get rid of preprocessing entirely, though for actually meaningless things like newlines / leading and following spaces you might as well remove them anyway.

To address your new info briefly...

Yes, character indexes for NER labels must be updated if you do preprocessing. If they aren't updated they aren't usable.

It looks like you're trying to extract "skills" from a resume. That has many bullet point lists. The spaCy training data is newspaper articles, which don't contain any lists like that, so it's hard to say what the right thing to do is. I don't think the bullets matter much, but you can try removing or not removing them.

What about stuff like lowercasing, stop words, lemmatizing, etc?

I already addressed this, but do not do this. This was historically common practice for NLP models, but for modern neural models, including spaCy, it is actively unhelpful.