Crawler Libraries - Page 12

Go spider

Go 121Updated: 3 y ago License: No License (No License)

B站用户爬虫 好耶~是爬虫

Python 120Updated: 4 y ago License: Permissive (MIT)

components search using component-crawler

JavaScript 120Updated: 4 y ago License: No License (No License)

This is a course-downloader to help NTU students download courses data from NTU Ceiba.

Python 120Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

Amazon S3 bucket finder and crawler.

Python 119Updated: 2 y ago License: Permissive (MIT)

制作自己的VOC2007数据集用于faster-rcnn目标检测模型训练

Python 119Updated: 4 y ago License: No License (No License)

NodeJS robots.txt parser with support for wildcard (*) matching.

JavaScript 119Updated: 2 y ago License: Permissive (MIT)

Take a snapshot of any website.

TypeScript 119Updated: 3 y ago License: Permissive (MIT)

Python爬虫和Flask实现小说网站

Python 119Updated: 4 y ago License: Permissive (MIT)

Golang爬虫 爬取豆瓣电影Top250

Go 119Updated: 4 y ago License: No License (No License)

yande.re图片爬虫

Python 118Updated: 1 y ago License: No License (No License)

一只百度文库的爬虫 A spider of baiduwenku

Python 118Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered.

Python 118Updated: 3 y ago License: Proprietary (Proprietary)

Python 118Updated: 3 y ago License: No License (No License)

Heart diseases are complicated and take away lots of life's every year.

R 118Updated: 1 y ago License: No License (No License)

🕷️用于爬取B站前top100的小视频

Python 117Updated: 4 y ago License: Permissive (MIT)

（已跑路）百度贴吧吧务管理工具，自动扫描帖子并处理违规帖

C++ 117Updated: 3 y ago License: Strong Copyleft (GPL-2.0)

[AppStore, PlayStore] 전국 노점상, 푸드트럭 지도 "가슴속3천원"

Kotlin 117Updated: 2 y ago License: No License (No License)

Sample of using proxies to crawl baidu search results.

Python 116Updated: 4 y ago License: No License (No License)

a spider for cnki patent content, just for study and commucation, no use for business.

Python 116Updated: 4 y ago License: No License (No License)

scrapy专利爬虫（停止维护）

Python 116Updated: 4 y ago License: Permissive (Apache-2.0)

Python distributed web scrapper and dynamic crawler

Python 116Updated: 4 y ago License: Permissive (MIT)

国家统计用区划代码和城乡划分代码---爬虫及数据

Python 116Updated: 4 y ago License: Permissive (Apache-2.0)

Spider of learning

Python 116Updated: 1 y ago License: No License (No License)

Extract encrypted Google Chrome cookies for a url on a Mac or Linux

JavaScript 116Updated: 2 y ago License: Permissive (MIT)

滑动验证码，希望对你们有所帮助❤️

Python 116Updated: 4 y ago License: No License (No License)

多线程爬虫--抓取淘宝商品详情页URL

Java 115Updated: 3 y ago License: No License (No License)

爬取电影天堂的电影爬虫

Python 115Updated: 3 y ago License: No License (No License)

This is what I do with Pthon distributed crawler

Python 115Updated: 4 y ago License: No License (No License)

A simple, fast, and reliable Coursera crawling & downloading tool

Python 115Updated: 4 y ago License: Permissive (MIT)

可能是java界最好的开源行为验证码 [滑块验证码、点选验证码、行为验证码、旋转验证码， 滑动验证码]

Java 115Updated: 2 y ago License: Proprietary (Proprietary)

Command-line Google dork tool. This is an early predecessor to dorkbot, which may be more useful: https://github.com/utiso/dorkbot

Python 114Updated: 4 y ago License: Permissive (BSD-2-Clause)

Scrape web data at scale.

HTML 114Updated: 3 y ago License: Strong Copyleft (AGPL-3.0)

Python爬虫和Flask实现小说网站

Python 114Updated: 4 y ago License: Permissive (MIT)

Varoius IDC-scripts I've collected during the years.

Python 113Updated: 4 y ago License: No License (No License)

:star2::octocat: powered by python3( simple learning of spider) 百度文库；网易云歌曲； 豆瓣电影； GitHub； 京东； QQ空间； 天气； vip解析助手； TED文本内容； wifi破解脚本； 必应图片设置为桌面等爬取

Python 113Updated: 4 y ago License: No License (No License)

Twitter crawler based on python

Python 113Updated: 3 y ago License: Permissive (MIT)

分享一些爬虫脚本

Python 113Updated: 4 y ago License: Permissive (MIT)

一只优雅的正方教务系统爬虫。

PHP 113Updated: 2 y ago License: Permissive (MIT)

Cross-platform persistent and distributed web crawler :link:

Go 113Updated: 4 y ago License: Permissive (MIT)

又一个 java 内容（pa）获取（chong）工具

Java 112Updated: 4 y ago License: Permissive (Apache-2.0)

Keyword extraction based on TF-IDF on specific corpus. 基于特定语料库的TF-IDF的中文关键词提取

Python 112Updated: 3 y ago License: Permissive (MIT)

Unmaintained :whale: :coffee: :spider: Scala crawler(spider) framework, inspired by scrapy, created by @gaocegege

Scala 112Updated: 4 y ago License: No License (No License)

lots of spider (很多爬虫）

Python 111Updated: 4 y ago License: No License (No License)

:spider:招聘网站爬虫合集，不定期更新分支

Python 111Updated: 3 y ago License: No License (No License)

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:

Python 111Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

A tool to crawl systems like crawlers for the web

Python 111Updated: 2 y ago License: Permissive (Apache-2.0)

Easy way to brute-force web directory.

Python 111Updated: 4 y ago License: Permissive (Unlicense)

Utility to crawl and diff websites for node.js

JavaScript 111Updated: 4 y ago License: Permissive (MIT)

Web of Science Crawler

Python 110Updated: 4 y ago License: No License (No License)