Crawler Libraries - Page 8

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Go 205Updated: 4 y ago License: Permissive (MIT)

Distributed redis-based web crawler framework for colly

Go 205Updated: 3 y ago License: No License (No License)

A python web crawler

Python 204Updated: 4 y ago License: No License (No License)

京东爬虫，可抓取京东商品信息和评论

Python 204Updated: 3 y ago License: No License (No License)

🍿爬虫代理IP池(proxy pool) python🍟一个还ok的IP代理池

Python 203Updated: 3 y ago License: Permissive (MIT)

Get (almost) original messages from google group archives. Your data is yours.

Shell 203Updated: 3 y ago License: No License (No License)

The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)

Java 201Updated: 4 y ago License: No License (No License)

以Node.js基于express以及爬虫实现的视频资源后端

JavaScript 201Updated: 3 y ago License: No License (No License)

golang light-weight image crawler

Go 201Updated: 4 y ago License: Permissive (Apache-2.0)

Astra is a tool to find URLs and secrets inside a webpage/files

Python 201Updated: 2 y ago License: No License (No License)

A fast and powerfull dashboard (admin) finder

Python 200Updated: 4 y ago License: Strong Copyleft (GPL-2.0)

golang spider Crawler 爬虫 电影

Go 199Updated: 2 y ago License: Permissive (MIT)

天猫双12爬虫，附商品数据。

Python 198Updated: 3 y ago License: No License (No License)

A set of reusable Java components that implement functionality common to any web crawler

Java 197Updated: 2 y ago License: Permissive (Apache-2.0)

book.zi5.me全站kindle电子书籍爬取，按照作者书籍名分类，每本书有mobi和equb两种格式，采用分布式进行全站爬取

Python 196Updated: 3 y ago License: No License (No License)

대량의 뉴스 데이터를 수집하기 위해 만들어진 뉴스 크롤러입니다.

Python 195Updated: 2 y ago License: Permissive (MIT)

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

Go 195Updated: 4 y ago License: Permissive (Apache-2.0)

Web template crawler written by golang. golang开发的网站模板爬虫.

Go 194Updated: 4 y ago License: No License (No License)

张俊的读书笔记

Jupyter Notebook 194Updated: 2 y ago License: Permissive (MIT)

不用找了。这里是全网最优解。免费且由个人开发，并且长期维护 python selenim QQ空间 虎牙 电影 微信 微博 抖音 知乎 B站 bili Bili 百度贴吧 小红书 tiktok youtube twitter 有颜色 爬虫

Python 194Updated: 1 y ago License: No License (No License)

a taobao web crawler just for fun.

Python 193Updated: 4 y ago License: Permissive (MIT)

News crawling with StormCrawler - stores content as WARC

Java 193Updated: 2 y ago License: Permissive (Apache-2.0)

超星学习通答题姬（视频文档观看、模拟答题，无需浏览器、无需油猴，容器/host 运行ok！

Python 193Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Download tencent comic-下载腾讯漫画

Python 192Updated: 4 y ago License: No License (No License)

⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

Go 192Updated: 1 y ago License: Permissive (Apache-2.0)

네이버 뉴스 수집을 위한 도구

R 190Updated: 3 y ago License: Proprietary (Proprietary)

91 porn crawler. 自动爬取并下载你想要的91porn热门视频。Automatically download your "favorite" 91porn hot movies.

Java 189Updated: 3 y ago License: No License (No License)

Web Site Page Changes Monitor. 网站网页页面更新变更监控提醒。

JavaScript 189Updated: 2 y ago License: Proprietary (Proprietary)

"Scrape Easy" - an extension of the Scrapy framework.

Python 188Updated: 4 y ago License: No License (No License)

A Distributed Crawler System Designed By Java.

Java 187Updated: 4 y ago License: Permissive (Apache-2.0)

JS逆向Hook工具集，开源部分工具到这里

JavaScript 187Updated: 1 y ago License: No License (No License)

My Python Script

Python 186Updated: 3 y ago License: Permissive (Apache-2.0)

爬爬爬

Python 186Updated: 4 y ago License: No License (No License)

基于 Selenium 的知乎关键词爬虫

JavaScript 186Updated: 4 y ago License: No License (No License)

QQ 空间动态爬虫，利用cookie登录获取所有可访问好友空间的动态保存到本地

Python 185Updated: 4 y ago License: No License (No License)

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

PHP 185Updated: 2 y ago License: Permissive (MIT)

🐝 Web vertical crawler framework for fun

Java 184Updated: 4 y ago License: Permissive (MIT)

Crawl instagram photos, posts and videos for download.

Ruby 184Updated: 4 y ago License: Permissive (MIT)

A download manager for Baidu Yun using Tampermonkey

TypeScript 184Updated: 3 y ago License: Permissive (MIT)

python3网络爬虫笔记与实战源码。记录python爬虫学习全程笔记、参考资料和常见错误，约40个爬取实例与思路解析，涵盖urllib、requests、bs4、jsonpath、re、 pytesseract、PIL等常用库的使用。

HTML 184Updated: 3 y ago License: No License (No License)

A simple distributed crawler for zhihu && data analysis

Python 183Updated: 3 y ago License: Strong Copyleft (GPL-2.0)

Digger is a powerful and flexible web crawler implemented by pure golang

Go 183Updated: 2 y ago License: Permissive (MIT)

设计精良的压缩包密码破解工具，具有自定义字典、导出字典、选择字典等功能。基于Python实现，支持多线程与多进程，不断完善中……

Python 182Updated: 1 y ago License: Weak Copyleft (LGPL-3.0)

使用代理调用github API爬去用户数据

Python 182Updated: 2 y ago License: No License (No License)

A loose framework for crawling and scraping web sites.

Ruby 182Updated: 3 y ago License: Permissive (MIT)

Totally legal and legit updater for Yuzu Early Access

C# 182Updated: 1 y ago License: Strong Copyleft (GPL-2.0)

Web crawler for zhihu.com

JavaScript 181Updated: 4 y ago License: No License (No License)

Web spider as a service, spider on serverless

JavaScript 180Updated: 3 y ago License: Permissive (MIT)

xvideos API library

JavaScript 178Updated: 2 y ago License: Permissive (BSD-3-Clause)

百度云（网盘）下载助手

C# 178Updated: 4 y ago License: Strong Copyleft (GPL-3.0)