Scraper Libraries - Page 3

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

Python 387Updated: 2 y ago License: Permissive (MIT)

Locally saves webpages to your hard disk with images, css, js & links as is.

Python 386Updated: 1 y ago License: Proprietary (Proprietary)

`scrape_linkedin` is a python package that allows you to scrape personal LinkedIn profiles & company pages - turning the data into structured json.

HTML 383Updated: 2 y ago License: Permissive (MIT)

A universal web-util for PHP.

PHP 382Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Instagram OSINT tool to export and analyse followers | following with their details

Python 377Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

COVID-19 Coronavirus data scraped from government and curated data sources.

HTML 372Updated: 3 y ago License: Permissive (BSD-2-Clause)

Scrape Instagram's API with Puppeteer

TypeScript 351Updated: 2 y ago License: Permissive (MIT)

OSINT Tool For Scraping Dark Websites

Python 342Updated: 1 y ago License: Permissive (MIT)

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.

TypeScript 337Updated: 1 y ago License: Permissive (MIT)

Library for extracting marketing attribution data from referrer URLs

Python 334Updated: 1 y ago License: No License (No License)

Scrape/Crawl article from any site automatically. Make any web page readable, no matter Chinese or English.

JavaScript 334Updated: 2 y ago License: No License (No License)

Social media scraping / data collection tool for the Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

Python 333Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

The unofficial HLTV Node.js API

TypeScript 324Updated: 2 y ago License: Permissive (MIT)

reborn of https://bitbucket.org/rflechner/scrapysharp

C# 317Updated: 1 y ago License: Permissive (MIT)

Web Scraper used to create Kaggle European Soccer database

HTML 311Updated: 2 y ago License: No License (No License)

A tool to scrape a Prometheus client and dump the result as JSON.

Go 306Updated: 2 y ago License: Permissive (Apache-2.0)

Be nice on the web

R 305Updated: 2 y ago License: Proprietary (Proprietary)

Python web scraping framework

Python 304Updated: 4 y ago License: No License (No License)

A simple to learn and use, yet powerful web scraping toolkit!

JavaScript 299Updated: 4 y ago License: Strong Copyleft (GPL-2.0)

NBA Stats API via Basketball Reference

HTML 297Updated: 3 y ago License: Permissive (MIT)

Scrape, standardize and share public meetings from local government websites

Python 294Updated: 2 y ago License: Permissive (MIT)

Lightweight web scraping toolkit for documents and structured data.

Python 290Updated: 2 y ago License: Permissive (MIT)

Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.

Ruby 290Updated: 4 y ago License: Permissive (MIT)

A neat way to trigger JS when media queries change. No jQuery required.

JavaScript 287Updated: 4 y ago License: No License (No License)

A universal package of scraper scripts for humans

Python 278Updated: 3 y ago License: Permissive (MIT)

An API to scrape American court websites for metadata.

HTML 275Updated: 2 y ago License: Permissive (BSD-2-Clause)

Python 272Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

Simple web scraping for Google Chrome.

JavaScript 270Updated: 4 y ago License: Permissive (BSD-3-Clause)

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

Kotlin 270Updated: 1 y ago License: Strong Copyleft (AGPL-3.0)

Simple cookie framework with full Unicode support

JavaScript 258Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

Lightweight package to query popular search engines and scrape for result titles, links and descriptions

Python 256Updated: 3 y ago License: No License (No License)

This PHP library enables you to scrape data from IMDB.com.

PHP 256Updated: 2 y ago License: Permissive (MIT)

Rawler is a tool that crawls the links of your website

Ruby 254Updated: 4 y ago License: Permissive (MIT)

A Python library for scraping the Google search engine.

Python 252Updated: 2 y ago License: Permissive (MIT)

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Python 247Updated: 4 y ago License: Permissive (ISC)

Scrape web data at scale completely and accurately with high performance, distributed RPA.

Kotlin 247Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

A Ruby DSL for structured web crawling, with a robust caching system.

Ruby 246Updated: 3 y ago License: Permissive (MIT)

Google Maps reviews scraping

Python 242Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Search on Google, and crawls for emails related to the result

Python 240Updated: 3 y ago License: No License (No License)

A simple browser/client-side web scraper.

TypeScript 238Updated: 4 y ago License: Permissive (MIT)

An Aggregator Engine for searching and downloading movies free - NO ADs!

Go 237Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

Generate link previews, inspired by Slack.

Kotlin 235Updated: 2 y ago License: Permissive (Apache-2.0)

A web scraper to retrieve application data from the Google Play Store.

Python 232Updated: 1 y ago License: Permissive (MIT)

SEC Edgar Scraper and XBRL Parser/Renderer

Python 232Updated: 3 y ago License: Permissive (MIT)

Jupyter Notebook 232Updated: 1 y ago License: Permissive (MIT)

Fetch user's data across social media

Python 230Updated: 2 y ago License: Permissive (Apache-2.0)

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

PHP 230Updated: 3 y ago License: Permissive (MIT)

A declarative struct-tag-based HTML unmarshaling or scraping package for Go built on top of the goquery library

Go 229Updated: 2 y ago License: Permissive (MIT)

A scraping command line tool for the modern web

JavaScript 226Updated: 4 y ago License: Permissive (MIT)

Python script that scrapes the currently trending YouTube videos in a variety of countries

Python 224Updated: 2 y ago License: Permissive (BSD-2-Clause)