Crawler Libraries

FILTER

LANGUAGES

All

LICENSES

All

COMPONENT TYPES

All

SUPPORT

All

SOURCES

All

SECURITY

All

INDUSTRIES

All
Click on the libraries for details

Sort by

Relevance
s

scrapyby scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Python Updated: 3 mo ago License: Proprietary

Support
Quality
Security
License
Reuse
c

cheerioby cheeriojs

Fast, flexible, and lean implementation of core jQuery designed specifically for the server.

TypeScript Updated: 6 d ago License: Permissive

Support
Quality
Security
License
Reuse
w

winstonby winstonjs

A logger for just about everything.

JavaScript Updated: 8 d ago License: Permissive

Support
Quality
Security
License
Reuse
p

pyspiderby binux

A Powerful Spider(Web Crawler) System in Python.

Python Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
c

collyby gocolly

Elegant Scraper and Crawler Framework for Golang

Go Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
p

python-spiderby Jack-Cherish

:rainbow:Python3网络爬虫实战:淘宝、京东、网易云、B站、12306、抖音、笔趣阁、漫画小说下载、音乐电影下载等

Python Updated: 6 mo ago License: No License

Support
Quality
Security
License
Reuse
p

proxy_poolby jhao104

Python爬虫代理IP池(proxy pool)

Python Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
F

FileDownloaderby lingochamp

Multitask、MultiThread(MultiConnection)、Breakpoint-resume、High-concurrency、Simple to use、Single/NotSingle-process

Java Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
e

examples-of-web-crawlersby shengqiangzhang

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Python Updated: 6 mo ago License: Permissive

Support
Quality
Security
License
Reuse
w

webmagicby code4craft

A scalable web crawler framework for Java.

Java Updated: 3 mo ago License: Permissive

Support
Quality
Security
License
Reuse
P

Photonby s0md3v

Incredibly fast crawler designed for OSINT.

Python Updated: 3 mo ago License: Strong Copyleft

Support
Quality
Security
License
Reuse
c

crawlabby crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Go Updated: 4 mo ago License: Permissive

Support
Quality
Security
License
Reuse
a

avbookby guyueyingmu

AV 电影管理系统, avmoo , javbus , javlibrary 爬虫,线上 AV 影片图书馆,AV 磁力链接数据库,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

PHP Updated: 6 mo ago License: No License

Support
Quality
Security
License
Reuse
P

Pythonby injetlee

Python脚本。模拟登录知乎, 爬虫,操作excel,微信公众号,远程开机

Python Updated: 6 mo ago License: No License

Support
Quality
Security
License
Reuse
p

pholcusby henrylee2cn

Pholcus is a distributed high-concurrency crawler software written in pure golang

Go Updated: 6 mo ago License: Permissive

Support
Quality
Security
License
Reuse
n

node-crawlerby bda-research

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

JavaScript Updated: 13 d ago License: Permissive

Support
Quality
Security
License
Reuse
I

InfoSpiderby kangvcar

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰,旨在安全快捷的帮助用户拿回自己的数据,工具代码开源,流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

Python Updated: 4 mo ago License: Strong Copyleft

Support
Quality
Security
License
Reuse
f

fuck-loginby xchaoinfo

模拟登录一些知名的网站,为了方便爬取需要登录的网站

Python Updated: 6 mo ago License: No License

Support
Quality
Security
License
Reuse
P

PythonSpiderNotesby lining0806

Python入门网络爬虫之精华版

Python Updated: 6 mo ago License: No License

Support
Quality
Security
License
Reuse
W

WechatSogouby chyroc

基于搜狗微信搜索的微信公众号爬虫接口

Python Updated: 5 mo ago License: Permissive

Support
Quality
Security
License
Reuse