Crawler Libraries - Page 11

一个五子棋学习资料站

HTML 137Updated: 2 y ago License: Permissive (MIT)

B站3亿用户信息爬虫（mid号，昵称，性别，关注，粉丝，等级）

Python 136Updated: 4 y ago License: Permissive (MIT)

A Simple spider that use to crawl the douban Top 100 moive name and input all list

Python 136Updated: 4 y ago License: No License (No License)

Cross domain getImageData work around and jQuery plugin

JavaScript 136Updated: 4 y ago License: No License (No License)

百度文库爬虫 Baidu Wenku Spider 百度文库下载器

Python 136Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

Distributed crawler, database and web frontend for public directories indexing

Python 135Updated: 1 y ago License: Permissive (MIT)

A spider/crawler edit by Node.js to download torrents of Adult videos.

JavaScript 134Updated: 3 y ago License: No License (No License)

a .js scanner, built in php. designed to scrape urls and other info

CSS 134Updated: 4 y ago License: No License (No License)

Argo is an automated general crawler for automatically obtaining website URLs . Argo 是一个自动化通用爬虫 用于自动化获取网站的URL 基于无头浏览器实现了静态和动态结合的方式来实现

Go 134Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

Github stargazers information gathering tool

Python 133Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

:star2::octocat: powered by python3( simple learning of spider) 百度文库；网易云歌曲； 豆瓣电影； GitHub； 京东； QQ空间； 天气； vip解析助手； TED文本内容； wifi破解脚本； 必应图片设置为桌面等爬取

Python 133Updated: 2 y ago License: No License (No License)

Immospider is a crawler for the Immoscout24 website.

Jupyter Notebook 133Updated: 2 y ago License: No License (No License)

Learn the web crawler.

Python 132Updated: 3 y ago License: No License (No License)

Zhihu User Spider

Python 132Updated: 4 y ago License: No License (No License)

An adult websites crawler for PPAV .

HTML 132Updated: 3 y ago License: Permissive (MIT)

Leetcode Contest Ranking Searcher

JavaScript 131Updated: 2 y ago License: Permissive (Apache-2.0)

字体混淆服务

Python 130Updated: 3 y ago License: Permissive (MIT)

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

Python 130Updated: 4 y ago License: No License (No License)

企业信息爬虫，关键字爬取公司信息

Python 130Updated: 4 y ago License: Permissive (MIT)

预约美帝签证各个签证处最早时间的爬虫

Python 130Updated: 4 y ago License: No License (No License)

scrapy-redis的集群版，可以借助Redis集群实现海量网站的独立去重，避免单机内存不足的尴尬

Python 130Updated: 4 y ago License: No License (No License)

爬取淘宝商品信息

Python 129Updated: 2 y ago License: No License (No License)

a web crawler

Python 129Updated: 3 y ago License: No License (No License)

备份豆瓣计划

Python 129Updated: 3 y ago License: Permissive (MIT)

Kubernetes Redis with High Availability

Shell 129Updated: 3 y ago License: Permissive (MIT)

react + flask + scrapy 构建的单页应用漫画网站

JavaScript 128Updated: 4 y ago License: No License (No License)

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Python 127Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

Weibo Spider Using Scrapy

Python 126Updated: 4 y ago License: No License (No License)

A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, sitemap-index, robots.txt, or arbitrary input

JavaScript 126Updated: 2 y ago License: Permissive (MIT)

⚡ Instantly search for the availability of your username on more than 100 social media sites.

JavaScript 125Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

用爬虫爬取小说网站上所有小说，存储到数据库中，并用爬到的数据构建自己的小说网站

CSS 125Updated: 4 y ago License: No License (No License)

A flask API for running your scrapy spiders

Python 124Updated: 4 y ago License: Proprietary (Proprietary)

Multithreading download all HD photos / pictures from someone's Sina Weibo album.

Python 124Updated: 4 y ago License: Permissive (Apache-2.0)

一个基于SVM的验证码破解程序

Python 124Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

Pixiv插画批量下载，提供关注画师插画、收藏作品下载（单/多/动图）及API - pixiv爬虫

Python 123Updated: 2 y ago License: Permissive (MIT)

pdd (拼多多) 爬虫 js 解密 anti_content 参数解密及全站抓取代码思路实现

Python 123Updated: 4 y ago License: No License (No License)

Distributed Ruby Web Crawler, backed up by Redis

Ruby 123Updated: 4 y ago License: No License (No License)

A crawler on taobao live barrages.

JavaScript 123Updated: 4 y ago License: Permissive (MIT)

The unix-way web crawler

Go 123Updated: 2 y ago License: Permissive (MIT)

新浪微博主题爬虫

Java 122Updated: 4 y ago License: No License (No License)

Scrape GSoC organisations using a single script.

Python 122Updated: 3 y ago License: Permissive (MIT)

Scrapy YouTube watch history spider. Because YouTube didn't have a history search.

Python 122Updated: 4 y ago License: Permissive (Unlicense)

Some scrapy spiders useful to crawl instagram posts using public APIS (No TOKEN)

Python 122Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure :dizzy: 多语言执行器，分布式爬虫

JavaScript 122Updated: 3 y ago License: Permissive (MIT)

文章采集工具 Article collection tool

Go 122Updated: 3 y ago License: Permissive (Apache-2.0)

Sometimes sites make crawling hard. Selenium-crawler uses selenium automation to fix that.

Python 121Updated: 4 y ago License: Permissive (MIT)

百度mp3全站爬虫

Python 121Updated: 3 y ago License: No License (No License)

爬虫-博客大全

JavaScript 121Updated: 4 y ago License: No License (No License)

A php crawler that finds emails on the internets

PHP 121Updated: 4 y ago License: Permissive (MIT)

微博爬虫：每天定时爬取微博热搜榜的内容，留下互联网人的记忆。

Python 121Updated: 3 y ago License: Strong Copyleft (GPL-3.0)