Crawler Libraries - Page 16

一个可以自动生成PTGen，MediaInfo/BDInfo，截图，并且生成发布所需内容的快速发种工具

Python 75Updated: 3 y ago License: Permissive (MIT)

fetchman is a simple crawler system/简单好用的爬虫框架

Python 74Updated: 4 y ago License: Permissive (Apache-2.0)

Some scrapy and web.py exmaples

Python 74Updated: 4 y ago License: No License (No License)

A DHT Spider allows you to sniff the torrents and magnets.You can download them directly.

Python 74Updated: 2 y ago License: Strong Copyleft (AGPL-3.0)

A crawler for submissions on leetcode-cn. 这是一个用来爬取力扣中国(LeetCode CN)提交代码的爬虫。

Python 74Updated: 1 y ago License: No License (No License)

A web crawler for Sina, search and retrieve microblogs that contain certain keywords 一个简单的python爬虫实践，爬取包含关键词的新浪微博

Python 74Updated: 3 y ago License: No License (No License)

Entry for js13k 2020

JavaScript 74Updated: 4 y ago License: Permissive (WTFPL)

基于go-zero实现的网盘系统

Go 74Updated: 2 y ago License: Permissive (MIT)

Golang短视频去水印：抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

Go 74Updated: 2 y ago License: No License (No License)

database meta crawl tool

Java 73Updated: 4 y ago License: No License (No License)

A compact, flexible Java multi-threaded crawler framework (Ai Pa), built-in Jsoup, zero-cost hands-on.一款小巧、灵活的Java多线程爬虫框架（AiPa）内嵌Jsoup 零成本上手

Java 73Updated: 4 y ago License: Permissive (Apache-2.0)

a simplified directed customizable website crawler

JavaScript 73Updated: 4 y ago License: Permissive (MIT)

HollyJS Moscow

JavaScript 73Updated: 3 y ago License: Permissive (MIT)

A simple Zoomeye written by python,more details click this link: http://blog.csdn.net/u011721501/article/details/41967847

PHP 73Updated: 4 y ago License: No License (No License)

Python爬虫：基础，进阶，框架, csdn,糗事百科,百度贴吧.淘宝MM ,豆瓣电影排行榜 ,腾讯招聘网站,斗鱼主播,汽车网站,百度学术, 必应学术,百科 ,金融实体关系,微博(用户,微博,评论,社交网络)，推特Twitter

Jupyter Notebook 73Updated: 3 y ago License: No License (No License)

a quick start python mutil thread crawl

Python 73Updated: 3 y ago License: Permissive (MIT)

分布式定向抓取集群

Python 72Updated: 5 y ago License: No License (No License)

Some classic web crawler projects.一些经典的爬虫

Python 72Updated: 4 y ago License: Permissive (Apache-2.0)

A simple search engine based on the web crawler developed in Udacity's CS101 course.

Python 72Updated: 4 y ago License: Proprietary (Proprietary)

收集新浪微博数据

Python 72Updated: 3 y ago License: No License (No License)

python crawler spider

Python 72Updated: 4 y ago License: No License (No License)

Scrapy spider example for Scrapy Tutorial Series

Python 72Updated: 4 y ago License: No License (No License)

Samurai Email Discovery - SED is a email discovery framework that grabs emails via google dork, company name, or domain name.

Shell 72Updated: 3 y ago License: No License (No License)

拉勾网爬虫 lagou spider

HTML 72Updated: 4 y ago License: No License (No License)

A crawler to collect reviews and product information on Amazon.com

Java 71Updated: 4 y ago License: Permissive (Unlicense)

playing around with the common crawl dataset

Java 71Updated: 4 y ago License: No License (No License)

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

Python 71Updated: 4 y ago License: Permissive (ISC)

微博爬虫：每天定时爬取微博热搜榜的内容，留下互联网人的记忆。

Python 71Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

A scrapy project can crawl search result of Google/Bing/Baidu

Python 71Updated: 4 y ago License: No License (No License)

MS17-010 multithreading scanner written in python.

Python 71Updated: 4 y ago License: No License (No License)

爬一些常用的电影网站的连接

Python 71Updated: 4 y ago License: No License (No License)

the scrapy_paper crawls the page file

HTML 71Updated: 3 y ago License: No License (No License)

Yet Another Web Spider

Python 70Updated: 4 y ago License: No License (No License)

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

Python 70Updated: 4 y ago License: No License (No License)

Your preferred open source focused crawler for the deep web.

Java 70Updated: 2 y ago License: Permissive (Apache-2.0)

Scrapy Selenium on Taobao Product

Python 70Updated: 4 y ago License: No License (No License)

QtWebKit-based web crawler

Python 70Updated: 4 y ago License: No License (No License)

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.

C# 70Updated: 4 y ago License: No License (No License)

Инструмент сбора данных о разделах, товарах и позициях товаров в разделах Wildberries и других российских маркетплейсов

Python 70Updated: 2 y ago License: No License (No License)

A simple, command-line based RSS enclosure downloader, primarily intended for automatic, unattended downloading of podcasts.

C 70Updated: 3 y ago License: Proprietary (Proprietary)

Multi-threaded implementation of redis written in rust

Rust 70Updated: 4 y ago License: Strong Copyleft (GPL-3.0)

利用urllib2加beautifulsoup爬取新浪微博

Python 69Updated: 5 y ago License: Permissive (Apache-2.0)

Python crawler for quora.com

Python 69Updated: 4 y ago License: No License (No License)

SimFin's open source PDF crawler

Python 69Updated: 3 y ago License: No License (No License)

A RabbitMQ Scheduler for Scrapy

Python 69Updated: 4 y ago License: Permissive (MIT)

自建免费IP代理池。

Python 69Updated: 4 y ago License: No License (No License)

Weighted PageRank implementation in Go

Go 69Updated: 4 y ago License: Permissive (MIT)

这其实是一份学习笔记。包括学习记录、爬虫练习平台（网站）、自制工具脚本

HTML 69Updated: 2 y ago License: Permissive (MIT)

Python爬虫：基础，进阶，框架, csdn,糗事百科,百度贴吧.淘宝MM ,豆瓣电影排行榜 ,腾讯招聘网站,斗鱼主播,汽车网站,百度学术, 必应学术,百科 ,金融实体关系,微博(用户,微博,评论,社交网络)，推特Twitter

Jupyter Notebook 69Updated: 3 y ago License: No License (No License)

知乎爬虫，基于webmagic框架 .A java web spider base on webmagic.

Java 68Updated: 4 y ago License: No License (No License)