named-entity-recognition | A simple NER model in C | Natural Language Processing library

This should be all you need:

I'm also following this tutorial. It worked for me without adding these tokens, however, I found in another tutorial (https://vamvas.ch/bert-for-ner) that it is better to add them, because the model was trained in this format.

[Update] Actually just checked it, it turned out that the accuracy improved by 20% after adding the tokens. But note that I am using it on a different dataset

The pipeline object can do that for you when you set the parameter grouped_entities to True.

you are missing the last layer activation:

This might not be a good application of NLP as the text isn't any sort of natural language. Rather, they are structured data that can be extracted using rules. Writing rules is definitely one way to go about this.

1. You can first try to do a fuzzy match of the categories on the OCR results, namely "CARDIAC RISK" and "CHEMISTRIES" to partition the string into their respective categories.

2. If you are sure that each entry will take only 3 lines, you can simply partition them by newline and extract the data from there.

3. Once you have them split into entries

Here's some sample code I ran on the data you provided. It requires the fuzzyset package which you can get by running python3 -m pip install fuzzyset. Since some entries don't have units I modified your desired output format slightly and made units a list so it can easily be empty. It also stores random letters found in the third line.

For n_sent=1

Turning the characters in the image of the pdf to text would be more of a computer vision task, and it seems like this is not what you're looking to do since you seem more interested in phrase extraction which would be NLP. Therefore the first step is probably to extract the text from the pdfs before feeding the text into NLP libraries for phrase extraction.

There seem to be a good number of libraries in Python to do pdf text extraction - this pops up from a quick Google search. As for the NLP, there are lots of libraries and concepts to learn in this field, again a quick Google search gets this article as an intro to NLP in Python.

Here is some code to get you going. It is not a complete solution, but the problem you posed is very hard, and you didn't have any starter code.

It does not track the identifier or NCBI Homologene properties, but I think those can be stored in a dictionary separately.

You can do this without quanteda, using the spacyr package -- a wrapper around the spaCy library mentioned in your linked article.

Here, I have slightly edited your input data.frame.