Crawler Libraries - Page 5

极客时间课程抓取脚本，支持输入账号密码后自动将极客时间的专栏课程保存到本地

Python 449Updated: 2 y ago License: Permissive (MIT)

a big hairy fuzzy spider that crawls your site, wreaking havoc

Ruby 449Updated: 4 y ago License: Permissive (MIT)

Html网页正文提取

C# 442Updated: 4 y ago License: Proprietary (Proprietary)

Scrapy Book Code

Python 427Updated: 3 y ago License: No License (No License)

Multi threading and processing eye-candy.

Python 422Updated: 2 y ago License: Weak Copyleft (LGPL-3.0)

各种App、小程序、网站的请求签名或加密算法。 现已有：自如、小红书、蛋壳公寓、luckin coffee(瑞幸咖啡)、bangkokair(曼谷航空)

Python 416Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

🎬 基于Pyqt5的简单电影搜索工具

Python 415Updated: 4 y ago License: Strong Copyleft (GPL-2.0)

PTT 網路版爬蟲

Python 409Updated: 2 y ago License: Permissive (MIT)

扫描“微信读书”已购图书并下载本地PDF的爬虫

Python 407Updated: 1 y ago License: No License (No License)

a smart stream-like crawler & etl python library

Python 404Updated: 4 y ago License: Permissive (Apache-2.0)

The Traditional Swiss Army Knife for OSINT

Python 401Updated: 4 y ago License: Strong Copyleft (GPL-2.0)

跨平台的京东全能工具包 仅供学习使用，技术交流群：108934299

Python 400Updated: 3 y ago License: No License (No License)

ACHE is a web crawler for domain-specific search.

Java 399Updated: 2 y ago License: Permissive (Apache-2.0)

高效微信公众号历史文章和阅读数据爬虫powered by scrapy

JavaScript 395Updated: 2 y ago License: No License (No License)

北京浮生记PC版源代码

C++ 392Updated: 3 y ago License: Strong Copyleft (GPL-2.0)

Videodl: A lightweight video downloader written by pure python.

Python 385Updated: 2 y ago License: Permissive (Apache-2.0)

A zhihu Spider.Just for fun.

PHP 380Updated: 4 y ago License: No License (No License)

🌈Python3网络爬虫实战：QQ音乐歌曲、京东商品信息、房天下、破解有道翻译、构建代理池、豆瓣读书、百度图片、破解网易登录、B站模拟扫码登录、小鹅通、荔枝微课

Python 377Updated: 2 y ago License: No License (No License)

The agile query language for semi-structured data

Python 369Updated: 2 y ago License: Permissive (MIT)

🔎 前程无忧 Python 招聘岗位信息爬取和分析

Python 369Updated: 2 y ago License: Permissive (MIT)

招聘网数据爬虫

HTML 367Updated: 2 y ago License: Permissive (MIT)

a tool for crawl Google search results

Python 365Updated: 1 y ago License: Permissive (MIT)

open source, distributed, restful crawler engine in golang

Go 365Updated: 4 y ago License: Permissive (MIT)

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

Python 361Updated: 3 y ago License: Strong Copyleft (AGPL-3.0)

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

Python 361Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

抖音爬虫

Python 354Updated: 2 y ago License: No License (No License)

golang实现的爬虫框架，使用者只需关心页面规则，提供web管理界面。基于colly开发。

Go 353Updated: 2 y ago License: Permissive (MIT)

今日头条机器人，支持用户登陆、关注、取消关注、获取关注粉丝、发文、发悟空问答、点赞、评论、采集各种类型新闻讯息等，使用今日头条网页版API实现

Python 352Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

知乎模拟登录，支持提取验证码和保存 Cookies

Python 347Updated: 4 y ago License: Permissive (MIT)

爬虫+数据分析实战项目

Jupyter Notebook 347Updated: 2 y ago License: No License (No License)

spider-admin-pro 一个集爬虫Scrapy+Scrapyd爬虫项目查看 和 爬虫任务定时调度的可视化管理工具，SpiderAdmin的升级版

Python 346Updated: 1 y ago License: Permissive (MIT)

一个爬虫式的网段Web主机发现小工具 # A HTTP Service detector with a crawler from IP/CIDR

Python 344Updated: 3 y ago License: No License (No License)

快速、简洁且强大的PHP爬虫框架

PHP 344Updated: 4 y ago License: No License (No License)

短视频的PHP拓展包，集成各大短视频的去水印功能、抖音、快手、微视主流短视频。PHP去水印

PHP 344Updated: 4 y ago License: Permissive (MIT)

Spider

Python 343Updated: 4 y ago License: Strong Copyleft (GPL-2.0)

🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览

JavaScript 343Updated: 2 y ago License: No License (No License)

长行的爬虫集合：微博、Twitter、玩加、知网、虎牙、斗鱼、B站、WeGame、猫眼、豆瓣、安居客、居理新房

Python 342Updated: 2 y ago License: No License (No License)

在线地址: http://119.23.223.90:8000

Python 342Updated: 3 y ago License: Permissive (MIT)

Super configurable async web spider

JavaScript 342Updated: 4 y ago License: No License (No License)

A scalable and convenient crawler framework in C:).

C 341Updated: 3 y ago License: Permissive (MIT)

爬取慕课网视频

Python 339Updated: 3 y ago License: No License (No License)

python 编写的DHT Crawler 网络爬虫，抓取磁力链接

Python 335Updated: 3 y ago License: No License (No License)

淘宝爬虫SDK，用于淘宝开放平台或淘宝、天猫、阿里巴巴登录爬取

Python 334Updated: 2 y ago License: Permissive (MIT)

Some crawlers u know it:-)

Python 332Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

Google search results crawler, get google search results that you need

Python 328Updated: 2 y ago License: No License (No License)

🕷spider world with me

Python 327Updated: 4 y ago License: No License (No License)

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

JavaScript 327Updated: 3 y ago License: Permissive (Apache-2.0)

漫画爬虫，支持腾讯漫画、哔哩哔哩漫画、有妖气漫画、快看漫画、漫画柜等主流漫画站点 ac.qq.com manga.bilibili.com u17.com kuaikanmanhua.com manhuagui.com. Comic Manga Crawler

Python 321Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

Cheerio for Google Apps Script

JavaScript 320Updated: 2 y ago License: Permissive (MIT)

微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。

Python 320Updated: 3 y ago License: No License (No License)