How to Remove Stop Words with NLTK in Python

by vigneshchennai74 Updated: Jan 19, 2023

Stop words are words that do not contain essential meanings and are usually removed from texts. They are words filtered out before or after processing natural language data. Stop words are commonly used words in any language, not just English. Examples of stop words include: a, an, and, the, of, or, in, on, at, etc.

To remove Stopwords using python:

Firstly, you need to have a list of stop words.
Then you can use the nltk library to tokenize the text and filter out the stop words.
nltk: The Natural Language Toolkit (nltk) is a library in Python that provides tools to work with human language data (text).
Alternatively, you can use the sklearn library to create a list of stop words and filter them out of the text.
Sklearn: This is a machine learning library for Python. It provides a wide range of tools for data preprocessing, classification, regression, clustering, model selection, and dimensionality reduction via a consistent interface.

Here is how you can remove stop words with NLTK in Python;

CODE

In this code we using NLTK library from python to Remove Stop word.

spam filtering: remove stopwords

PythonLines of Code : 22License : CC BY-SA 4.0

import nltk
data = [['ham', 'And how you will do that, princess? :)'], ['spam', 'Urgent! Please call 09061213237 from landline. £5000 cash or a luxury 4* Canary Islands Holiday await collection']]

for text in (label_text[1] for label_text in data): 
    filtered_tokens = [token for token in nltk.word_tokenize(text) if token.lower() not in nltk.corpus.stopwords.words('english')]
    print(filtered_tokens)

>>> [',', 'princess', '?', ':', ')']
>>> ['Urgent', '!', 'Please', 'call', '09061213237', 'landline', '.', '£5000', 'cash', 'luxury', '4*', 'Canary', 'Islands', 'Holiday', 'await', 'collection']

Copy the code using "Copy" and paste it in your python ide,
check whether nltk library is added.
Enter the data that need to remove Stopwords
Run the code and get the Output

I hope you found this useful. I have added the link to dependent libraries, version information in the following sections.

I found this code snippet by searching for "Spam filtering: remove Stopwords" in kandi. You can try any such use case!

Dependent Libraries

nltkby nltk

Python 11448 Version:3.8.1

License: Permissive (Apache-2.0)

NLTK Source

Support

Quality

Security

License

Reuse

nltkby nltk

Python 11448 Version:3.8.1 License: Permissive (Apache-2.0)

NLTK Source

Support

Quality

Security

License

Reuse

Environment Tested

I tested this solution in the following versions. Be mindful of changes when working with other versions

The solution is created and executed in Python version 3.7.15

Using this solution, we are able to remove the stopwords in Python with simple steps. This process also facilities an easy to use, hassle free method to create a hands-on working version of code which would help us remove words in Python..