Back ArrowExplore all topics
technology logo
technology logo

How to Turn spaCy doc into Nested List of Tokens

share link

by vigneshchennai74 dot icon Updated: Feb 1, 2023

Solution Kit Solution Kit  

The spaCy library provides the Doc object to represent a document, which can be tokenized into individual words or phrases (tokens) using the “doc.sents” and doc[i] attributes. You can convert a Doc object into a nested list of tokens by iterating through the sentences in the document, and then iterating through the tokens in each sentence. 

  • spaCy: spaCy is a library for advanced natural language processing in Python. It is designed specifically for production use, and it is fast and efficient. spaCy is widely used for natural language processing tasks such as named entity recognition, part-of-speech tagging, text classification, and others. 
  • Doc.sents: Doc.sents allows you to work with individual sentences easily and efficiently in a text, rather than having to manually split the text into sentences yourself. This can be useful in a variety of natural languages processing tasks, such as sentiment analysis or text summarization, where it's important to be able to work with individual sentences. 


To learn more about the topic, you may have a look at the code below

Preview of the output that you will get on running this code from your IDE

Code

In this solution we used spaCy library of python.

How to turn spacy doc into nested list of tokens

PythondotLines of Code : 37dotLicense : Strong Copyleft (CC BY-SA 4.0)

Dependent Libraries : 
import spacy

# Load relevant language/pipeline: here, the built-in small English web-based
# model.
nlp = spacy.load("en_core_web_sm")

# Run text through pipeline to create annotated doc.
sample_text = "Colorless green ideas sleep furiously."
doc = nlp(sample_text)

# Iterate through each token (t) in the doc object, and create a nested list
# of the children of each token. Keep in mind that like many spaCy attributes,
# token.children returns a generator. To access all of its elements at once,
# you will have to convert this generator into an object of type list.
child_list = [list(t.children) for t in doc]

# Now as an exercise, print out each token and check to see if you get the
# children you expected. Normally you would want to iterate on the objects 
# themselves -- we only use range() here for purposes of illustration.
for i in range(len(doc)):
    print("  token {}: {}".format(i + 1, doc[i]))
    print("    children: {}\n".format(child_list[i]))

$ python example.py
  token 1: Colorless
    children: []
  token 2: green
    children: []
  token 3: ideas
    children: [Colorless, green]
  token 4: sleep
    children: [ideas, furiously, .]
  token 5: furiously
    children: []
  token 6: .
    children: []
  1. Copy the code using the "Copy" button above, and paste it in a Python file in your IDE.
  2. Import Sapcy library
  3. Run the file to turn spacy doc into nested List of tokens.


I hope you found this useful. I have added the link to dependent libraries, version information in the following sections.


I found this code snippet by searching for "How to turn spacy doc into nested list of tokens"in kandi. You can try any such use case.

Environment Tested

I tested this solution in the following versions. Be mindful of changes when working with other versions.


  1. The solution is created in Python 3.7.15 Version
  2. The solution is tested on Spacy 3.4.3 Version


Using this solution, we can turn the spacy doc into nested list in tokens with the help of function in spacy . This process also facilities an easy to use, hassle free method to create a hands-on working version of code which would help us turn the doc to nestled list in the text in python.

Dependent Library

spaCyby explosion

Python doticonstar image 26205 doticonVersion:v3.5.3doticon
License: Permissive (MIT)

💫 Industrial-strength Natural Language Processing (NLP) in Python

Support
    Quality
      Security
        License
          Reuse

            spaCyby explosion

            Python doticon star image 26205 doticonVersion:v3.5.3doticon License: Permissive (MIT)

            💫 Industrial-strength Natural Language Processing (NLP) in Python
            Support
              Quality
                Security
                  License
                    Reuse

                      If you do not have SpaCy that is required to run this code, you can install it by clicking on the above link and copying the pip Install command from the Spacy page in kandi.

                      You can search for any dependent library on kandi like Spacy

                      Support

                      1. For any support on kandi solution kits, please use the chat
                      2. For further learning resources, visit the Open Weaver Community learning page

                      See similar Kits and Libraries

                      Open Weaver – Develop Applications Faster with Open Source

                      kandi

                      Community and Support

                      Company

                      Follow

                      • © 2023 Open Weaver Inc.