Crawler Libraries - Page 4

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos视频爬取，有声书爬取，微博爬虫，安居客信息爬取+数据可视化，哔哩哔哩视频封面提取器，ip代理池封装，知乎百万级用户爬虫+数据分析，github用户爬虫

Python 623Updated: 1 y ago License: Permissive (MIT)

An implementation of a Reflexion agent for SOTA Human-Eval Python results.

Python 621Updated: 2 y ago License: Permissive (MIT)

A Facebook crawler

Python 619Updated: 2 y ago License: Permissive (Apache-2.0)

a spider on the apps

Java 615Updated: 3 y ago License: No License (No License)

crawl and scrape web pages in rust

Rust 614Updated: 1 y ago License: Permissive (Apache-2.0)

htcap is a web application scanner able to crawl single page application (SPA) recursively by intercepting ajax calls and DOM changes.

Python 600Updated: 2 y ago License: Strong Copyleft (GPL-2.0)

豆瓣电影/豆瓣读书 Scarpy 爬虫

Python 597Updated: 1 y ago License: No License (No License)

📖Java工程师学习历程与笔记，附含算法、Java基础、框架实战、框架源码、框架实现、中间件、面试题等知识和学习蓝图。

Java 597Updated: 2 y ago License: Permissive (MIT)

短视频的PHP拓展包，集成各大短视频的去水印功能、抖音、快手、微视主流短视频。PHP去水印

PHP 597Updated: 1 y ago License: Permissive (MIT)

扒网站工具，看好哪个网站，指定好URL，自动扒下来做成模版。所见网站，皆可为我所用！

Java 596Updated: 1 y ago License: Permissive (Apache-2.0)

新手小白~~持续更新中〜随意Fork，Star，大佬萌多多issue哦 (目前更新: 1.短视频视频无水印解析（支持超过30多个平台 e.g., 抖音，快手，微博...） 2.bilibili音视频解析🐱‍👓 3.知音漫客图片加密 🔥 4. 世纪佳缘信息👨‍👩‍👧‍👦 5.小鸡词典🐥🐥。。。。。。

Python 592Updated: 1 y ago License: No License (No License)

API of DouYin for Humans used to Crawl Popular Videos and Musics

Python 587Updated: 3 y ago License: Permissive (MIT)

Python爬虫，返回一定格式的信息，下载，使用flask提供简易api。抖音无水印、皮皮虾、快手、网易云音乐、qq音乐、咪咕音乐、荔枝FM音频、知乎视频、最右语音、视频、微博......

Python 585Updated: 2 y ago License: Permissive (MIT)

JS破解逆向，破解JS反爬虫加密参数，已破解极验滑块w（2022.2.19），QQ音乐sign（2022.2.13），拼多多anti_content，boss直聘zp_token，知乎x-zse-96，酷狗kg_mid/dfid，唯品会mars_cid，中国裁判文书网（2020-06-30更新），淘宝密码，天安保险登录，b站登录，房天下登录，WPS登录，微博登录，有道翻译，网易登录，微信公众号登录，空中网登录，今目标登录，学生信息管理系统登录，共赢金融登录，重庆科技资源共享平台登录，网易云音乐下载，一键解析视频链接，财联社登录。

JavaScript 581Updated: 2 y ago License: No License (No License)

NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。

Java 579Updated: 4 y ago License: Permissive (Apache-2.0)

This is a collection of stuff that I wrote for Proxmox.

Shell 579Updated: 2 y ago License: Permissive (MIT)

brozzler - distributed browser-based web crawler

Python 578Updated: 2 y ago License: Permissive (Apache-2.0)

最简单的91porn爬虫php版本

PHP 573Updated: 3 y ago License: No License (No License)

Calibre new douban metadata source plugin. Douban no longer provides book APIs to the public, so it can only use web crawling to obtain data. This is a calibre Douban plugin based on web crawling.

Python 572Updated: 1 y ago License: Permissive (Apache-2.0)

python爬虫，包含大小项目

Python 571Updated: 2 y ago License: No License (No License)

抓取网易云音乐热门评论

Python 569Updated: 3 y ago License: Permissive (MIT)

爬取菜鸟教程网站并转PDF__python_crawer_by_chrome

Python 565Updated: 2 y ago License: No License (No License)

crawl webm and mp4

Python 546Updated: 2 y ago License: No License (No License)

Headless Chrome For Java （Java 爬虫）

Java 545Updated: 1 y ago License: Permissive (Apache-2.0)

带你了解一下Golang的市场行情

Go 538Updated: 3 y ago License: No License (No License)

Python的基础练习代码与各种爬虫代码

Python 537Updated: 2 y ago License: No License (No License)

📺 B 站全站视频信息爬虫

Python 537Updated: 2 y ago License: Permissive (MIT)

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Go 518Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

crawl webm and mp4

Python 516Updated: 3 y ago License: No License (No License)

Simple PHP Proxy: Get external HTML, JSON and more!

JavaScript 513Updated: 4 y ago License: Proprietary (Proprietary)

Crawler for Nintendo Switch eShop

TypeScript 510Updated: 2 y ago License: Permissive (Apache-2.0)

抖音下载【无水印】的视频（2023-01-08更新，兼容android 11+）

Java 509Updated: 1 y ago License: No License (No License)

a new crawler based on python with more function including Network fingerprint search

C 506Updated: 4 y ago License: No License (No License)

crawl webm and mp4

Python 501Updated: 3 y ago License: No License (No License)

Cookies Pool

Python 500Updated: 3 y ago License: No License (No License)

一款 Chrome 扩展。用于备份豆瓣账号的数据，并支持导出 Excel 文档。

JavaScript 497Updated: 2 y ago License: Permissive (MIT)

Python interface to Boilerpipe, Boilerplate Removal and Fulltext Extraction from HTML pages

Python 494Updated: 3 y ago License: Proprietary (Proprietary)

Wget-compatible web downloader and crawler.

HTML 491Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

🎬 基于Pyqt5的简单电影搜索工具

Python 483Updated: 3 y ago License: Strong Copyleft (GPL-2.0)

天眼查爬虫&企查查爬虫，指定关键字爬取公司信息

Python 476Updated: 2 y ago License: Permissive (MIT)

QQ空间爬虫，可导出并显示日志、相册、留言板、说说、照片、视频等数据。

Python 473Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

crawl webm and mp4

Python 471Updated: 4 y ago License: No License (No License)

A framework for creating semi-automatic web content extractors

Python 470Updated: 4 y ago License: Permissive (MIT)

一个功能全面的微信读书笔记助手 wereader

Python 466Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

QQZone mood spider and analysis. QQ空间多线程爬虫和数据挖掘。提供线上服务，扫码登陆即可自动爬取和分析数据，还有网易云年度报告风格的数据展示；使用docker-compose打包程序，方便部署；额外提供QQ空间抽奖小程序。

Python 461Updated: 3 y ago License: Permissive (MIT)

👩 美女写真套图爬虫（一）

Python 461Updated: 2 y ago License: Permissive (MIT)

汤不热 python 多线程爬虫

Python 460Updated: 3 y ago License: Permissive (MIT)

a reliable high-level web crawling & scraping framework for Node.js.

JavaScript 453Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

Simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.

Go 450Updated: 3 y ago License: Permissive (MIT)

When jsoup meets XPath.

Java 449Updated: 1 y ago License: Permissive (MIT)