Back ArrowExplore all topics
technology logo
technology logo

Modify Tokenize for numeric patterns

share link

by vigneshchennai74 dot icon Updated: Jan 1, 2023

Solution Kit Solution Kit  

In this solution we are going to Modify tokenizer for numeric patterns using spacy. Spacy is the most prominent and useful library in python it consist of numerous numbers of functions that help the developer to develop easily .In that solution we used spaCy - Util.compile_infix_regex function .This utility function will compile a sequence of infix rules into a regex object. In this solution kit, I am sharing the code snippet and library that I use to remove particular tokes in Python which can be executed directly in the IDE.


Preview of the output that you will get on running this code from your IDE

Code

In this solution we have used tokenizer function.

Spacy - modify tokenizer for numeric patterns

PythondotLines of Code : 37dotLicense : Strong Copyleft (CC BY-SA 4.0)

Dependent Libraries : 
import spacy
from spacy.lang.char_classes import ALPHA, ALPHA_LOWER, ALPHA_UPPER, HYPHENS
from spacy.lang.char_classes import CONCAT_QUOTES, LIST_ELLIPSES, LIST_ICONS
from spacy.util import compile_infix_regex

# default tokenizer
nlp = spacy.load("en_core_web_sm")
doc = nlp("111-222-1234 for abcDE")
print([t.text for t in doc])

# modify tokenizer infix patterns
infixes = (
        LIST_ELLIPSES
        + LIST_ICONS
        + [
            r"(?<=[0-9])[+\*^](?=[0-9-])", # Remove the hyphen
            r"(?<=[{al}{q}])\.?(?=[{au}{q}])".format( # Make the dot optional
                al=ALPHA_LOWER, au=ALPHA_UPPER, q=CONCAT_QUOTES
            )
            ,
            r"(?<=[{a}]),(?=[{a}])".format(a=ALPHA),
            r"(?<=[{a}])(?:{h})(?=[{a}])".format(a=ALPHA, h=HYPHENS),
            r"(?<=[{a}0-9])[:<>=/](?=[{a}])".format(a=ALPHA),
        ]
)

infix_re = compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_re.finditer
doc = nlp("111-222-1234 for abcDE")
print([t.text for t in doc])

With default tokenizer:
['111', '-', '222', '-', '1234', 'for', 'abcDE']

With custom tokenizer:
['111-222-1234', 'for', 'abc', 'DE']
  1. Copy the code using the "Copy" button above, and paste it in a Python file in your IDE.
  2. Enter the Text
  3. Run the file to get custom Tokenized Numeric patterns


I hope you found this useful. I have added the link to dependent libraries, version information in the following sections.


I found this code snippet by searching for "Modify tokenize for Numeric pattern " in kandi. You can try any such use case!

Dependent Library

spaCyby explosion

Python doticonstar image 25599 doticonVersion:v3.5.1doticon
License: Permissive (MIT)

💫 Industrial-strength Natural Language Processing (NLP) in Python

Support
    Quality
      Security
        License
          Reuse

            spaCyby explosion

            Python doticon star image 25599 doticonVersion:v3.5.1doticon License: Permissive (MIT)

            💫 Industrial-strength Natural Language Processing (NLP) in Python
            Support
              Quality
                Security
                  License
                    Reuse

                      If you do not have SpaCy that is required to run this code, you can install it by clicking on the above link and copying the pip Install command from the Spacy page in kandi.

                      You can search for any dependent library on kandi like Spac

                      Environment Test

                      I tested this solution in the following versions. Be mindful of changes when working with other versions.


                      1. The solution is created in Python 3.7.15 Version
                      2. The solution is tested on Spacy 3.4.3 Version


                      Using this solution, we can modify tokenizer for numeric pattern with the help of function in spacy . This process also facilities an easy to use, hassle free method to create a hands-on working version of code which would help us modify the numeric pattern in python

                      Support

                      1. For any support on kandi solution kits, please use the chat
                      2. For further learning resources, visit the Open Weaver Community learning page

                      Open Weaver – Develop Applications Faster with Open Source

                      kandi

                      Community and Support

                      Company

                      Follow

                      • © 2023 Open Weaver Inc.