Crawler Libraries - Page 2

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

Java 2444Updated: 2 y ago License: Permissive (MIT)

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Python 2405Updated: 1 y ago License: No License (No License)

一个用于安装/更新 NS 模拟器的工具

Python 2363Updated: 1 y ago License: Strong Copyleft (AGPL-3.0)

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

PHP 2354Updated: 1 y ago License: Permissive (MIT)

Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等

Go 2126Updated: 2 y ago License: Permissive (MIT)

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

C# 2071Updated: 2 y ago License: Permissive (Apache-2.0)

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Go 2047Updated: 1 y ago License: Weak Copyleft (MPL-2.0)

Web crawling framework based on asyncio.

Python 2016Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API

Python 2011Updated: 2 y ago License: Permissive (MIT)

Gospider - Fast web spider written in Go

Go 1993Updated: 1 y ago License: Permissive (MIT)

突破百度云限速合集，另外附带Baidu-Go、Tampermonkey、Proxyee-down教程。从此云端女友从不断线，有了这个它，忘掉那个她!

JavaScript 1956Updated: 2 y ago License: No License (No License)

Polite, slim and concurrent web crawler.

Go 1938Updated: 3 y ago License: Permissive (BSD-3-Clause)

一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.

Java 1904Updated: 1 y ago License: Permissive (Apache-2.0)

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

Python 1839Updated: 1 y ago License: Proprietary (Proprietary)

哔哩哔哩的API调用模块

Python 1830Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Python 1817Updated: 2 y ago License: Permissive (MIT)

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Go 1807Updated: 2 y ago License: Weak Copyleft (MPL-2.0)

🔎Sniffing and parsing mysql,redis,http,mongodb etc protocol. 抓包截取项目中的数据库请求并解析成相应的语句。

Go 1798Updated: 1 y ago License: Permissive (MIT)

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

PHP 1779Updated: 1 y ago License: Permissive (MIT)

简单易用的Python爬虫框架，QQ交流群：597510560

Python 1766Updated: 2 y ago License: Permissive (BSD-2-Clause)

Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.

Python 1707Updated: 2 y ago License: Permissive (MIT)

Async Python 3.6+ web scraping micro-framework based on asyncio

Python 1680Updated: 1 y ago License: Permissive (Apache-2.0)

Anemone web-spider framework

Ruby 1615Updated: 2 y ago License: Permissive (MIT)

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

JavaScript 1561Updated: 2 y ago License: No License (No License)

EditThisCookie is the famous Google Chrome/Chromium extension for editing cookies

JavaScript 1549Updated: 2 y ago License: Proprietary (Proprietary)

Crawl a website and run it through Google lighthouse

JavaScript 1470Updated: 2 y ago License: Permissive (ISC)

🏀 Python3 网络爬虫实战（部分含详细教程）猫眼 腾讯视频 豆瓣 研招网 微博 笔趣阁小说 百度热点 B站 CSDN 网易云阅读 阿里文学 百度股票 今日头条 微信公众号 网易云音乐 拉勾 有道 unsplash 实习僧 汽车之家 英雄联盟盒子 大众点评 链家 LPL赛程 台风 梦幻西游、阴阳师藏宝阁 天气 牛客网 百度文库 睡前故事 知乎 Wish

Python 1455Updated: 1 y ago License: No License (No License)

A configurable and extensible PHP web spider

PHP 1291Updated: 2 y ago License: Permissive (MIT)

Python爬虫，京东自动登录，在线抢购商品

Python 1269Updated: 1 y ago License: No License (No License)

微博爬虫及配套工具箱，微博用户、话题、评论采集一网打尽。图片下载、情感分析，地理位置、关系网络、spammer 机器人识别等功能应有尽有。Docs：https://buyixiao.github.io/blog/weibo-super-spider.html 配套可视化网站：https://buyixiao.github.io/blog/one-stop-weibo-visualization.html

Python 1218Updated: 2 y ago License: Permissive (Apache-2.0)

QQ空间导出助手，用于备份QQ空间的说说、日志、私密日记、相册、视频、留言板、QQ好友、收藏夹、分享、最近访客为文件，便于迁移与保存

JavaScript 1210Updated: 1 y ago License: Permissive (Apache-2.0)

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

PHP 1206Updated: 1 y ago License: Permissive (MIT)

Elasticsearch File System Crawler (FS Crawler)

Java 1197Updated: 2 y ago License: Permissive (Apache-2.0)

Python爬虫，京东自动登录，在线抢购商品

Python 1197Updated: 3 y ago License: No License (No License)

The complete web scraping toolkit for PHP.

PHP 1188Updated: 1 y ago License: No License (No License)

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

Python 1146Updated: 1 y ago License: No License (No License)

⚡️Lightning-fast async download tool for bilibili and more | 快如闪电的异步下载工具，支持bilibili及更多

Python 1141Updated: 1 y ago License: Permissive (Apache-2.0)

A web privacy measurement framework

Python 1137Updated: 3 y ago License: Proprietary (Proprietary)

Creating Scrapy scrapers via the Django admin interface

Python 1121Updated: 1 y ago License: Permissive (BSD-3-Clause)

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Python 1114Updated: 2 y ago License: Permissive (MIT)

百度云网盘搜索引擎，包含爬虫 & 网站

JavaScript 1052Updated: 2 y ago License: No License (No License)

JSpider会每周更新至少一个网站的JS解密方式，欢迎 Star，交流微信：13298307816

JavaScript 1038Updated: 1 y ago License: No License (No License)

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Python 1027Updated: 1 y ago License: Permissive (MIT)

​:rocket: 采集|免费|优质|的订阅链接；科学上网，从娃娃抓起！

Python 1023Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

👧 美女写真套图爬虫（二）

Python 1012Updated: 2 y ago License: No License (No License)

A task based API for taking screenshots and scraping text from websites.

JavaScript 990Updated: 4 y ago License: Permissive (Apache-2.0)

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Python 983Updated: 2 y ago License: Proprietary (Proprietary)

哔哩哔哩的API调用模块

Python 959Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

A configurable web spider with a easy-to-use web console

Java 958Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Ruby 952Updated: 2 y ago License: Permissive (MIT)