List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)
Support
Quality
Security
License
Reuse
Locally saves webpages to your hard disk with images, css, js & links as is.
Support
Quality
Security
License
Reuse
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
Support
Quality
Security
License
Reuse
A universal web-util for PHP.
Support
Quality
Security
License
Reuse
Instagram OSINT tool to export and analyse followers | following with their details
Support
Quality
Security
License
Reuse
COVID-19 Coronavirus data scraped from government and curated data sources.
Support
Quality
Security
License
Reuse
Scrape Instagram's API with Puppeteer
Support
Quality
Security
License
Reuse
OSINT Tool For Scraping Dark Websites
Support
Quality
Security
License
Reuse
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Support
Quality
Security
License
Reuse
r
referer-parserby snowplow-referer-parser
Python 
334 Version:Current
License: No License (No License)
License: No License (No License)Library for extracting marketing attribution data from referrer URLs
Support
Quality
Security
License
Reuse
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
Support
Quality
Security
License
Reuse
Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Support
Quality
Security
License
Reuse
The unofficial HLTV Node.js API
Support
Quality
Security
License
Reuse
reborn of https://bitbucket.org/rflechner/scrapysharp
Support
Quality
Security
License
Reuse
Web Scraper used to create Kaggle European Soccer database
Support
Quality
Security
License
Reuse
A tool to scrape a Prometheus client and dump the result as JSON.
Support
Quality
Security
License
Reuse
Be nice on the web
Support
Quality
Security
License
Reuse
Python web scraping framework
Support
Quality
Security
License
Reuse
A simple to learn and use, yet powerful web scraping toolkit!
Support
Quality
Security
License
Reuse
NBA Stats API via Basketball Reference
Support
Quality
Security
License
Reuse
Scrape, standardize and share public meetings from local government websites
Support
Quality
Security
License
Reuse
Lightweight web scraping toolkit for documents and structured data.
Support
Quality
Security
License
Reuse
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Support
Quality
Security
License
Reuse
A neat way to trigger JS when media queries change. No jQuery required.
Support
Quality
Security
License
Reuse
A universal package of scraper scripts for humans
Support
Quality
Security
License
Reuse
An API to scrape American court websites for metadata.
Support
Quality
Security
License
Reuse
Support
Quality
Security
License
Reuse
Simple web scraping for Google Chrome.
Support
Quality
Security
License
Reuse
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
Support
Quality
Security
License
Reuse
Simple cookie framework with full Unicode support
Support
Quality
Security
License
Reuse
Lightweight package to query popular search engines and scrape for result titles, links and descriptions
Support
Quality
Security
License
Reuse
This PHP library enables you to scrape data from IMDB.com.
Support
Quality
Security
License
Reuse
Rawler is a tool that crawls the links of your website
Support
Quality
Security
License
Reuse
A Python library for scraping the Google search engine.
Support
Quality
Security
License
Reuse
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Support
Quality
Security
License
Reuse
Scrape web data at scale completely and accurately with high performance, distributed RPA.
Support
Quality
Security
License
Reuse
A Ruby DSL for structured web crawling, with a robust caching system.
Support
Quality
Security
License
Reuse
Google Maps reviews scraping
Support
Quality
Security
License
Reuse
Search on Google, and crawls for emails related to the result
Support
Quality
Security
License
Reuse
A simple browser/client-side web scraper.
Support
Quality
Security
License
Reuse
An Aggregator Engine for searching and downloading movies free - NO ADs!
Support
Quality
Security
License
Reuse
Generate link previews, inspired by Slack.
Support
Quality
Security
License
Reuse
A web scraper to retrieve application data from the Google Play Store.
Support
Quality
Security
License
Reuse
SEC Edgar Scraper and XBRL Parser/Renderer
Support
Quality
Security
License
Reuse
Support
Quality
Security
License
Reuse
s
social-media-profile-scrapersby shaikhsajid1111
Python 230 Version:Current License: Permissive (Apache-2.0)
Fetch user's data across social media
Support
Quality
Security
License
Reuse
Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
Support
Quality
Security
License
Reuse
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
Support
Quality
Security
License
Reuse
A scraping command line tool for the modern web
Support
Quality
Security
License
Reuse
Python script that scrapes the currently trending YouTube videos in a variety of countries
Support
Quality
Security
License
Reuse
L
List-of-user-agentsby tamimibrahim17
List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)
Python 387Updated: 2 y ago License: Permissive (MIT)
387Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
p
pywebcopyby rajatomar788
Locally saves webpages to your hard disk with images, css, js & links as is.
Python 386Updated: 2 y ago License: Proprietary (Proprietary)
386Updated: 2 y ago License: Proprietary (Proprietary)Support
Quality
Security
License
Reuse
s
scrape-linkedin-seleniumby austinoboyle
`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.
HTML 383Updated: 2 y ago License: Permissive (MIT)
383Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
P
PHPScraperby spekulatius
A universal web-util for PHP.
PHP 382Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
382Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
s
sterraxcylby novitae
Instagram OSINT tool to export and analyse followers | following with their details
Python 377Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
377Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
c
coronadatascraperby covidatlas
COVID-19 Coronavirus data scraped from government and curated data sources.
HTML 372Updated: 4 y ago License: Permissive (BSD-2-Clause)
372Updated: 4 y ago License: Permissive (BSD-2-Clause)Support
Quality
Security
License
Reuse
i
instamancerby ScriptSmith
Scrape Instagram's API with Puppeteer
TypeScript 351Updated: 2 y ago License: Permissive (MIT)
351Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
D
DarkScrapeby itsmehacker
OSINT Tool For Scraping Dark Websites
Python 342Updated: 2 y ago License: Permissive (MIT)
342Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
l
linkedin-profile-scraperby jvandenaardweg
🕵️♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
TypeScript 337Updated: 2 y ago License: Permissive (MIT)
337Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
r
referer-parserby snowplow-referer-parser
Library for extracting marketing attribution data from referrer URLs
Python 334Updated: 2 y ago License: No License (No License)
334Updated: 2 y ago License: No License (No License)Support
Quality
Security
License
Reuse
n
node-readabilityby Tjatse
Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.
JavaScript 334Updated: 2 y ago License: No License (No License)
334Updated: 2 y ago License: No License (No License)Support
Quality
Security
License
Reuse
r
reaperby ScriptSmith
Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs
Python 333Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
333Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
H
HLTVby gigobyte
The unofficial HLTV Node.js API
TypeScript 324Updated: 2 y ago License: Permissive (MIT)
324Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
S
ScrapySharpby rflechner
reborn of https://bitbucket.org/rflechner/scrapysharp
C# 317Updated: 2 y ago License: Permissive (MIT)
317Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
f
football-data-collectionby hugomathien
Web Scraper used to create Kaggle European Soccer database
HTML 311Updated: 2 y ago License: No License (No License)
311Updated: 2 y ago License: No License (No License)Support
Quality
Security
License
Reuse
p
prom2jsonby prometheus
A tool to scrape a Prometheus client and dump the result as JSON.
Go 306Updated: 2 y ago License: Permissive (Apache-2.0)
306Updated: 2 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
p
Support
Quality
Security
License
Reuse
c
cyborgby orf
Python web scraping framework
Python 304Updated: 4 y ago License: No License (No License)
304Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
s
scrubytby scrubber
A simple to learn and use, yet powerful web scraping toolkit!
JavaScript 299Updated: 4 y ago License: Strong Copyleft (GPL-2.0)
299Updated: 4 y ago License: Strong Copyleft (GPL-2.0)Support
Quality
Security
License
Reuse
b
basketball_reference_web_scraperby jaebradley
NBA Stats API via Basketball Reference
HTML 297Updated: 3 y ago License: Permissive (MIT)
297Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
c
city-scrapersby City-Bureau
Scrape, standardize and share public meetings from local government websites
Python 294Updated: 2 y ago License: Permissive (MIT)
294Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
m
memoriousby alephdata
Lightweight web scraping toolkit for documents and structured data.
Python 290Updated: 2 y ago License: Permissive (MIT)
290Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
w
webinspectorby davidesantangelo
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.
Ruby 290Updated: 4 y ago License: Permissive (MIT)
290Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
o
on-media-queryby JoshBarr
A neat way to trigger JS when media queries change. No jQuery required.
JavaScript 287Updated: 4 y ago License: No License (No License)
287Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
S
Scraperaby DarshanDeshpande
A universal package of scraper scripts for humans
Python 278Updated: 4 y ago License: Permissive (MIT)
278Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
j
juriscraperby freelawproject
An API to scrape American court websites for metadata.
HTML 275Updated: 2 y ago License: Permissive (BSD-2-Clause)
275Updated: 2 y ago License: Permissive (BSD-2-Clause)Support
Quality
Security
License
Reuse
P
Prowlby nettitude
Python 272Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
272Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
s
scraperby mnmldave
Simple web scraping for Google Chrome.
JavaScript 270Updated: 4 y ago License: Permissive (BSD-3-Clause)
270Updated: 4 y ago License: Permissive (BSD-3-Clause)Support
Quality
Security
License
Reuse
P
PulsarRPAby platonai
Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.
Kotlin 270Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)
270Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)Support
Quality
Security
License
Reuse
c
cookies.jsby madmurphy
Simple cookie framework with full Unicode support
JavaScript 258Updated: 4 y ago License: Strong Copyleft (GPL-3.0)
258Updated: 4 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
s
search-engine-parserby bisoncorps
Lightweight package to query popular search engines and scrape for result titles, links and descriptions
Python 256Updated: 4 y ago License: No License (No License)
256Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
P
PHP-IMDB-Grabberby FabianBeiner
This PHP library enables you to scrape data from IMDB.com.
PHP 256Updated: 2 y ago License: Permissive (MIT)
256Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
r
rawlerby oscardelben
Rawler is a tool that crawls the links of your website
Ruby 254Updated: 4 y ago License: Permissive (MIT)
254Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
g
googlesearchby Nv7-GitHub
A Python library for scraping the Google search engine.
Python 252Updated: 2 y ago License: Permissive (MIT)
252Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
w
wayback-machine-scraperby sangaline
A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.
Python 247Updated: 4 y ago License: Permissive (ISC)
247Updated: 4 y ago License: Permissive (ISC)Support
Quality
Security
License
Reuse
p
pulsarRPAby platonai
Scrape web data at scale completely and accurately with high performance, distributed RPA.
Kotlin 247Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)
247Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)Support
Quality
Security
License
Reuse
s
sinewby gurgeous
A Ruby DSL for structured web crawling, with a robust caching system.
Ruby 246Updated: 4 y ago License: Permissive (MIT)
246Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
g
googlemaps-scraperby gaspa93
Google Maps reviews scraping
Python 242Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
242Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
p
python-email-crawlerby samwize
Search on Google, and crawls for emails related to the result
Python 240Updated: 4 y ago License: No License (No License)
240Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
g
getsyby epiqueras
A simple browser/client-side web scraper.
TypeScript 238Updated: 4 y ago License: Permissive (MIT)
238Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
g
gophieby Go-phie
An Aggregator Engine for searching and downloading movies free - NO ADs!
Go 237Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)
237Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)Support
Quality
Security
License
Reuse
u
unfurlby saket
Generate link previews, inspired by Slack.
Kotlin 235Updated: 2 y ago License: Permissive (Apache-2.0)
235Updated: 2 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
p
play-scraperby danieliu
A web scraper to retrieve application data from the Google Play Store.
Python 232Updated: 2 y ago License: Permissive (MIT)
232Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
S
ScraXBRLby tooksoi
SEC Edgar Scraper and XBRL Parser/Renderer
Python 232Updated: 3 y ago License: Permissive (MIT)
232Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
t
tradingview-scraperby rushic24
Jupyter Notebook 232Updated: 2 y ago License: Permissive (MIT)
232Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
s
social-media-profile-scrapersby shaikhsajid1111
Fetch user's data across social media
Python 230Updated: 2 y ago License: Permissive (Apache-2.0)
230Updated: 2 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
a
arachnidby zrashwani
Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
PHP 230Updated: 4 y ago License: Permissive (MIT)
230Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
g
goqby andrewstuart
A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library
Go 229Updated: 2 y ago License: Permissive (MIT)
229Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
q
quickscrapeby ContentMine
A scraping command line tool for the modern web
JavaScript 226Updated: 4 y ago License: Permissive (MIT)
226Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
T
Trending-YouTube-Scraperby mitchelljy
Python script that scrapes the currently trending YouTube videos in a variety of countries
Python 224Updated: 2 y ago License: Permissive (BSD-2-Clause)
224Updated: 2 y ago License: Permissive (BSD-2-Clause)Support
Quality
Security
License
Reuse