Crawler Libraries - Page 10

WEIBO_SCRAPY is a Multi-Threading SINA WEIBO data extraction Framework in Python.

Python 154Updated: 4 y ago License: No License (No License)

.NET based webcrawler

C# 154Updated: 2 y ago License: Permissive (Apache-2.0)

A reference implementation in python of a simple crawler for Ads.txt

Python 154Updated: 2 y ago License: No License (No License)

Crawler for linguistic corpora

Python 153Updated: 3 y ago License: Proprietary (Proprietary)

Enjoy driving on a Javascriptive (originally Pythonic) way to Japanese AV!

JavaScript 153Updated: 3 y ago License: Permissive (Apache-2.0)

简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。

C# 153Updated: 4 y ago License: Permissive (Apache-2.0)

:blush::blush::blush: 知乎问题爬虫

Java 152Updated: 4 y ago License: Permissive (MIT)

CoCrawler is a versatile web crawler built using modern tools and concurrency.

Python 152Updated: 3 y ago License: Permissive (Apache-2.0)

A lite distributed Java spider framework :-)

Java 151Updated: 4 y ago License: Permissive (Apache-2.0)

python爬虫 全球网址URL滚动提取

Python 151Updated: 3 y ago License: No License (No License)

Python 151Updated: 4 y ago License: No License (No License)

netease-music-spider is a sipder that you can find beautiful girlfriend or handsome boyfriend.

Python 150Updated: 2 y ago License: No License (No License)

A scrapy-based Hacker News crawler.

Python 150Updated: 4 y ago License: Permissive (MIT)

all kinds of scrapy demo

Python 150Updated: 2 y ago License: No License (No License)

Nodejs Crawler, including schedule, spider, web ui config, proxy modules. using nodejs, redis/ssdb, hbase, phantomjs. css selector extraction rules and regex extraction rules supported.

JavaScript 150Updated: 3 y ago License: Proprietary (Proprietary)

由Python编写的全异步实现的动漫之家(dmzj)漫画批量下载器（爬虫）

Python 148Updated: 3 y ago License: Permissive (MIT)

The python crawler which automatically crawls the original microblogs and pictures of the specified user, analyzes the microblogs, and displays them in the form of html charts.

HTML 147Updated: 4 y ago License: No License (No License)

Redis Priority Queue offers a priority/timeline based queue for use with Redis

Python 146Updated: 2 y ago License: Weak Copyleft (LGPL-2.1)

Scrape a public LinkedIn profile.

Python 146Updated: 3 y ago License: Permissive (MIT)

Interactive CLI Web Crawler

Go 146Updated: 2 y ago License: Strong Copyleft (GPL-3.0)

使用scrapy和pandas完成对知乎300w用户的数据分析。首先使用scrapy爬取知乎网的300w，用户资料，最后使用pandas对数据进行过滤，找出想要的知乎大牛，并用图表的形式可视化。

HTML 146Updated: 4 y ago License: No License (No License)

Go process used to crawl websites

Go 146Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

html+ python +django +爬虫 +pyecharts 实时疫情动态

JavaScript 146Updated: 2 y ago License: No License (No License)

scrapy爬取知乎用户数据

Python 145Updated: 4 y ago License: No License (No License)

爬虫项目：链家网（普通/scrapy）、虎扑、维基百科、百度地图api、房天下（分布式爬虫）、微信公众号（代理池爬取）

Python 145Updated: 3 y ago License: No License (No License)

An Extensible Image Crawler

Python 145Updated: 4 y ago License: Permissive (MIT)

Everybody can be scrapy guru

JavaScript 145Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

🔥 Shadowsocks 账号爬虫

Python 144Updated: 4 y ago License: No License (No License)

📱 百度贴吧多线程扫码登陆 / 自动签到 / 自动打码

Python 144Updated: 3 y ago License: Permissive (MIT)

基于Python3的pornhub网站爬虫

Python 144Updated: 2 y ago License: Permissive (MIT)

This is a crawler for Sina Weiqun website(WAP) information, including given Weiqun's posts, replies, users and their follow relation. Written in Python 2.7.1, store data in SQLite3. Relation-crawling part customized on Github Project sina_reptile.

Python 144Updated: 4 y ago License: No License (No License)

A fast tool to fetch URLs from HTML attributes by crawl-in.

Shell 144Updated: 3 y ago License: Permissive (MIT)

Download a large list of files concurrently

Go 144Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

Scala DSL for web crawling

Scala 144Updated: 4 y ago License: Proprietary (Proprietary)

A distributed Sina Weibo Search spider base on Scrapy and Redis.

Python 143Updated: 4 y ago License: No License (No License)

爬取豆瓣小组帖子的爬虫。

Python 143Updated: 1 y ago License: No License (No License)

🌟⏳🌟 各种网站的签到（停止维护）

Python 143Updated: 3 y ago License: No License (No License)

国家自然科学基金查询

Python 143Updated: 2 y ago License: Proprietary (Proprietary)

(2020年最新)斗鱼弹幕抓取及可视化管理平台第二版，提供弹幕抓取、弹幕实时发送速度可视化、抓取记录查询、弹幕下载、自定义关键词统计、铁粉统计、高光时刻自动捕获、高频弹幕词云等功能，起飞~~~

TypeScript 143Updated: 2 y ago License: No License (No License)

Android wav/pcm 录音机，支持暂停、再录制。支持跳过静音区模式。

Java 141Updated: 1 y ago License: No License (No License)

一个基于✨HOOK机制的微信机器人，支持🌱安全新闻定时推送【FreeBuf，先知，安全客，奇安信攻防社区】，👯后缀名查询，⚡备案查询，⚡手机号归属地查询，⚡WHOIS信息查询，🎉星座查询，⚡天气查询，🌱摸鱼日历⚡微步威胁情报查询， 🐛美女视频，⚡美女图片，👯帮助菜单。📫 支持积分功能，😄自定义程度丰富，小白也可轻松上手！

Python 141Updated: 1 y ago License: Strong Copyleft (GPL-3.0)

爬虫所需要的IP代理，抓取九个网站的代理IP检测/清洗/入库/更新，添加调用接口

Python 140Updated: 3 y ago License: No License (No License)

As you can see, a kuaishou crawler

Python 140Updated: 3 y ago License: Permissive (MIT)

A Python crawler uses Facebook Graph API to crawling fan page's public posts, comments, and reactions.

Python 140Updated: 3 y ago License: Permissive (MIT)

A parallel web spider of PornHub.成人网站Pornhub的并行网络爬虫。

HTML 140Updated: 3 y ago License: Strong Copyleft (GPL-3.0)

useful crawler project for practice

Go 140Updated: 4 y ago License: No License (No License)

短视频图集图片去水印：快手,皮皮虾,最右,小红书,微博

PHP 140Updated: 2 y ago License: No License (No License)

MM131网站图片爬取 :rotating_light:

Python 137Updated: 3 y ago License: No License (No License)

An object crawler/property search library.

JavaScript 137Updated: 4 y ago License: Permissive (MIT)

Asynchronous Web Crawler & Scraper

Ruby 137Updated: 4 y ago License: Proprietary (Proprietary)