Crawler Libraries - Page 1

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python 47503Updated: 1 y ago License: Permissive (BSD-3-Clause)

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

TypeScript 26488Updated: 1 y ago License: Permissive (MIT)

A logger for just about everything.

JavaScript 20628Updated: 1 y ago License: Permissive (MIT)

Elegant Scraper and Crawler Framework for Golang

Go 19706Updated: 1 y ago License: Permissive (Apache-2.0)

Python爬虫代理IP池(proxy pool)

Python 18050Updated: 1 y ago License: Permissive (MIT)

:rainbow:Python3网络爬虫实战：淘宝、京东、网易云、B站、12306、抖音、笔趣阁、漫画小说下载、音乐电影下载等

Python 16227Updated: 1 y ago License: No License (No License)

A Powerful Spider(Web Crawler) System in Python.

Python 15891Updated: 1 y ago License: Permissive (Apache-2.0)

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Python 12136Updated: 1 y ago License: Permissive (MIT)

A scalable web crawler framework for Java.

Java 10861Updated: 1 y ago License: Permissive (Apache-2.0)

Multitask、MultiThread(MultiConnection)、Breakpoint-resume、High-concurrency、Simple to use、Single/NotSingle-process

Java 10805Updated: 2 y ago License: Permissive (Apache-2.0)

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Go 9884Updated: 1 y ago License: Permissive (BSD-3-Clause)

Incredibly fast crawler designed for OSINT.

Python 9703Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

PHP 8923Updated: 1 y ago License: No License (No License)

🕵️‍♂️ Collect a dossier on a person by username from thousands of sites

Python 8607Updated: 1 y ago License: Permissive (MIT)

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Python 8377Updated: 1 y ago License: No License (No License)

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

Java 8064Updated: 1 y ago License: Permissive (MIT)

Pholcus is a distributed high-concurrency crawler software written in pure golang

Go 7391Updated: 1 y ago License: Permissive (Apache-2.0)

新浪微博爬虫，用python爬取新浪微博数据

Python 6919Updated: 1 y ago License: No License (No License)

Pholcus is a distributed high-concurrency crawler software written in pure golang

Go 6819Updated: 3 y ago License: Permissive (Apache-2.0)

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

Python 6681Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

JavaScript 6422Updated: 2 y ago License: Permissive (MIT)

Python入门网络爬虫之精华版

Python 6183Updated: 1 y ago License: No License (No License)

模拟登录一些知名的网站，为了方便爬取需要登录的网站

Python 5791Updated: 1 y ago License: No License (No License)

基于搜狗微信搜索的微信公众号爬虫接口

Python 5517Updated: 1 y ago License: Permissive (Apache-2.0)

Distributed crawler powered by Headless Chrome

JavaScript 5368Updated: 2 y ago License: Permissive (MIT)

Redis-based components for Scrapy.

Python 5279Updated: 2 y ago License: Permissive (MIT)

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

Python 5238Updated: 2 y ago License: Permissive (MIT)

A Tool for Domain Flyovers

Go 5159Updated: 1 y ago License: Permissive (MIT)

:zap: A distributed crawler for weibo, building with celery and requests.

Python 4769Updated: 2 y ago License: Permissive (MIT)

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

Go 4502Updated: 2 y ago License: Permissive (Apache-2.0)

Open Source Web Crawler for Java

Java 4391Updated: 2 y ago License: Permissive (Apache-2.0)

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

Python 3941Updated: 1 y ago License: Permissive (MIT)

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Go 3768Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

C# 3664Updated: 1 y ago License: Permissive (MIT)

《我用爬虫一天时间“偷了”知乎一百万用户，只为证明PHP是世界上最好的语言 》所使用的程序

PHP 3497Updated: 1 y ago License: No License (No License)

新浪微博爬虫（Scrapy、Redis）

Python 3209Updated: 2 y ago License: No License (No License)

novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、充值订阅、新闻发布等功能。

Java 3084Updated: 1 y ago License: Permissive (Apache-2.0)

Collection of China illegal cases about web crawler 本项目用来整理所有中国大陆爬虫开发者涉诉与违规相关的新闻、资料与法律法规。致力于帮助在中国大陆工作的爬虫行业从业者了解我国相关法律，避免触碰数据合规红线。 [AD]中文知识图谱门户

HTML 3056Updated: 2 y ago License: No License (No License)

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Python 2993Updated: 1 y ago License: Permissive (MIT)

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

Java 2975Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

Python 2959Updated: 2 y ago License: No License (No License)

🔞 JAVClub - 让你的大姐姐不再走丢

JavaScript 2798Updated: 2 y ago License: Permissive (MIT)

Standards compliant HTML filter written in PHP

PHP 2785Updated: 1 y ago License: Weak Copyleft (LGPL-2.1)

An advanced web directory & file scanning tool that will be more powerful than DirBuster, Dirsearch, cansina, and Yu Jian.一个高级web目录、文件扫描工具，功能将会强于DirBuster、Dirsearch、cansina、御剑。

Python 2716Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

admin ui for scrapy/open source scrapinghub

Python 2640Updated: 2 y ago License: No License (No License)

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

PHP 2548Updated: 2 y ago License: No License (No License)

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Java 2485Updated: 1 y ago License: Proprietary (Proprietary)

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

Java 2477Updated: 3 y ago License: Permissive (Apache-2.0)

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Python 2461Updated: 1 y ago License: No License (No License)

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Python 2446Updated: 1 y ago License: No License (No License)