13 best Python Web Scraping libraries in 2021

by naveen.kumar@openweaver.com

Scrape information from webpage, use complex rules, extract to easy to use worksheets and more with these Python libraries.Get ratings, code snippets & documentation for each library.

Use the open source, cloud APIs, or public libraries listed below in your application development based on your technology preferences, such as primary language. The below list also provides a view of the components' rating on different dimensions such as community support availability, security vulnerability, and overall quality, helping you make an informed choice for implementation and maintenance of your application. Please review the components carefully, having a no license alert or proprietary license, and use them appropriately in your applications. Please check the component page for the exact license of the component. You can also get information on the component's features, installation steps, top code snippets, and top community discussions on the component details page. The links to package managers are listed for download, where packages are readily available. Otherwise, build from the respective repositories for use in your application. You can also use the source code from the repositories in your applications based on the respective license types.
y

you-getby soimort

:arrow_double_down: Dumb downloader that scrapes the web

Python Updated: 3 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
s

scrapyby scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python Updated: 3 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
r

requests-htmlby psf

Pythonic HTML Parsing for Humans™

Python Updated: 2 d ago License: Permissive

Support
Quality
Security
License
Reuse
n

newspaperby codelucas

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Python Updated: 3 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
p

portiaby scrapinghub

Visual scraping for Scrapy

Python Updated: 6 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
p

patternby clips

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Python Updated: 6 mo ago License: Permissive

Support
Quality
Security
License
Reuse
a

autoscraperby alirezamika

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Python Updated: 6 mo ago License: Permissive

Support
Quality
Security
License
Reuse
t

tweets_analyzerby x0rz

Tweets metadata scraper & activity analyzer

Python Updated: 6 mo ago License: Strong Copyleft

Support
Quality
Security
License
Reuse
r

ruiaby howie6879

Async Python 3.6+ web scraping micro-framework based on asyncio

Python Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
g

gdomby syrusakbary

DOM Traversing and Scraping using GraphQL

Python Updated: 7 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
s

scrapy-clusterby istresearch

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Python Updated: 6 mo ago License: Permissive

Support
Quality
Security
License
Reuse
o

openstates-scrapersby openstates

source for Open States scrapers

Python Updated: 2 d ago License: Strong Copyleft

Support
Quality
Security
License
Reuse
g

gazpachoby maxhumber

🥫 The simple, fast, and modern web scraping library

Python Updated: 7 mo ago License: Permissive

Support
Quality
Security
License
Reuse
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items

Find more libraries on