How to Remove tokens Like Symbols -Punctuation-Numbers with SpaCy in Python

share link

by vigneshchennai74 dot icon Updated: Jan 31, 2023

technology logo
technology logo

Solution Kit Solution Kit Β 

SpaCy is an open-source software library for advanced natural language processing. It assists you in creating programs that process and "understand" massive amounts of text because it was created expressly for use in production environments. The quick and effective tokenization offered by spaCy is one of its main advantages. SpaCy is frequently used for tasks including information extraction, machine translation, named entity recognition, part-of-speech tagging, and text summarization in business, academia, and government research projects. 


Additionally, spaCy offers tools for standard tasks like text classification, language recognition, working with word vectors and similarity, and more. You can use spaCy's tokenizer to remove certain types of tokens from a text.β€―You may use SpaCy in a few ways to get rid of tokens in text, including symbols, punctuation, and numerals. Some examples include: 

  • Eliminating common stop words: SpaCy has a built-in list of terms you can eliminate from your writing, like "and," "or," and "the." 
  • Eliminating punctuation: You may verify whether a token is a punctuation using the spacy.tokens.token.is punct property and then deletes it from the text. 
  • Removing numbers: To determine whether a token is a number and to delete it from the text, use the spacy.tokens.token.like num property. 
  • Removing symbols: To determine whether a token is a symbol or not and to delete it from the text, use the spacy.tokens.token.isalpha and spacy.tokens.token.is digit properties. 


Here is how you can remove tokens like symbols, punctuation, and numbers in SpaCy:

Preview of the output that you will get on running this code from your IDE

Code

In this solution we use the Attributes method of the SpaCy library.

  1. Copy the code using the "Copy" button above, and paste it in a Python file in your IDE.
  2. Enter the Text
  3. Run the file to annihilate symbols ,numbers and punctuation


I hope you found this useful. I have added the link to dependent libraries, version information in the following sections.


I found this code snippet by searching for "How to filter tokens from spacy Document " in kandi. You can try any such use case!


Note


In this snippet we are using a Language model (en_core_web_sm)

  1. Download the model using the command python -m spacy download en_core_web_sm .
  2. paste it in your terminal and download it.


Check the user's spacy version using pip show spacy command in users terminal.

  1. if its version 3.0 or above you will need to load it using nlp = spacy.load("en_core_web_sm")
  2. if its version is less than 3.0 you will need to load it using nlp = spacy.load("en")

Environment Test

I tested this solution in the following versions. Be mindful of changes when working with other versions.


  1. The solution is created in Python 3.7.15. Version
  2. The solution is tested on Spacy 3.4.3 Version
  3. The solution is tested on numpy 1.21.6 Version


Using this solution, we can able to delete or remove symbols ,punctuation, numbers using python with the help of Spacy library. This process also facilities an easy to use, hassle free method to create a hands-on working version of code which would help us remove the token in python.

Dependent Library

numpyby numpy

Python doticonstar image 23755 doticonVersion:v1.25.0rc1doticon
License: Permissive (BSD-3-Clause)

The fundamental package for scientific computing with Python.

Support
    Quality
      Security
        License
          Reuse

            numpyby numpy

            Python doticon star image 23755 doticonVersion:v1.25.0rc1doticon License: Permissive (BSD-3-Clause)

            The fundamental package for scientific computing with Python.
            Support
              Quality
                Security
                  License
                    Reuse

                      spaCyby explosion

                      Python doticonstar image 26383 doticonVersion:v3.2.6doticon
                      License: Permissive (MIT)

                      πŸ’« Industrial-strength Natural Language Processing (NLP) in Python

                      Support
                        Quality
                          Security
                            License
                              Reuse

                                spaCyby explosion

                                Python doticon star image 26383 doticonVersion:v3.2.6doticon License: Permissive (MIT)

                                πŸ’« Industrial-strength Natural Language Processing (NLP) in Python
                                Support
                                  Quality
                                    Security
                                      License
                                        Reuse

                                          If you do not have SpaCy and numpy that is required to run this code, you can install it by clicking on the above link and copying the pip Install command from the Spacy page in kandi.

                                          You can search for any dependent library on kandi like Spacy and numpy

                                          Support

                                          1. For any support on kandi solution kits, please use the chat
                                          2. For further learning resources, visit the Open Weaver Community learning page.

                                          See similar Kits and Libraries