下载 Bilibili 视频/番剧/电影/纪录片 等资源
Support
Quality
Security
License
Reuse
A complete automated financial news crawler built on the top of Scrapy framework.
Support
Quality
Security
License
Reuse
A spider on Dcard. Strong and speedy.
Support
Quality
Security
License
Reuse
:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
Support
Quality
Security
License
Reuse
Scrapy 爬虫框架教程源码
Support
Quality
Security
License
Reuse
Search against any element for standardized and default styles from all major rendering engines (WebKit, Blink, Gecko, Trident).
Support
Quality
Security
License
Reuse
Multithreaded Web spider crawler written in Rust.
Support
Quality
Security
License
Reuse
T
Targeted_Literature_Reviews_via_webscrapingby paulamartingonzalez
Jupyter Notebook 
85 Version:Current
License: No License (No License)
License: No License (No License)Web scraping to get articles for a given query. It returns an spreadsheet with titles, abstracts, doi and references of the article
Support
Quality
Security
License
Reuse
🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
Support
Quality
Security
License
Reuse
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
Support
Quality
Security
License
Reuse
tumblr解析网站
Support
Quality
Security
License
Reuse
Web Crawler
Support
Quality
Security
License
Reuse
Scrape the Google search result with Scrapy.
Support
Quality
Security
License
Reuse
Fast, highly configurable, cloud native dark web crawler.
Support
Quality
Security
License
Reuse
Scrape data from Goodreads using Scrapy and Selenium :books:
Support
Quality
Security
License
Reuse
Project thu thập điểm chuẩn đại học 2014 - 2018 và phân tích dữ liệu
Support
Quality
Security
License
Reuse
s
scrapy_enterprise_architectureby zhengwen09
Python 82 Version:Current License: No License (No License)
License: No License (No License)python scrapy 企业级分布式爬虫开发架构模板
Support
Quality
Security
License
Reuse
🎙 A collection of podcasts around the web.
Support
Quality
Security
License
Reuse
Python网络爬虫实战--红薯中文网、企名片、汽车之家、有道翻译、知乎
Support
Quality
Security
License
Reuse
a tiny downloader with console panel.
Support
Quality
Security
License
Reuse
Crawl and validate proxies from Internet
Support
Quality
Security
License
Reuse
Tool to automatic leak information using Hacking with engine searches
Support
Quality
Security
License
Reuse
A news crawler for BBC News, Reuters and New York Times.
Support
Quality
Security
License
Reuse
Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!
Support
Quality
Security
License
Reuse
NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider
Support
Quality
Security
License
Reuse
Dynamic configurable crawl (动态可配置化爬虫)
Support
Quality
Security
License
Reuse
Sougou Weixin Spider Using Proxy
Support
Quality
Security
License
Reuse
A spider library of several data sources.
Support
Quality
Security
License
Reuse
OLX Scraper in Python Scrapy
Support
Quality
Security
License
Reuse
A scrapy zhihu crawler
Support
Quality
Security
License
Reuse
[DEPRECATED] Add an entire threaded comment system to your Rails application with only 7 lines of code.
Support
Quality
Security
License
Reuse
This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own video player.
Support
Quality
Security
License
Reuse
🌉 基于Go+Vue实现的openLDAP后台管理项目
Support
Quality
Security
License
Reuse
large-scale user information crawler of zhihu
Support
Quality
Security
License
Reuse
Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.
Support
Quality
Security
License
Reuse
A package to get list of user agents based on filters such as operating system, software name etc..
Support
Quality
Security
License
Reuse
recruit 招聘爬虫+数据分析 1.爬虫： 采用Scrapy 分布式爬虫技术，使用mongodb作为数据存储，爬取的网站Demo为51job，数据我目前爬了有几千条 2.数据处理： 采用pandas对爬取的数据进行清洗和处理 2.数据分析： 采用flask后端获取mongodb数据，前端使用bootstrap3.echarts以及D3的词云图，如果喜欢请star or Fork,预览详见
Support
Quality
Security
License
Reuse
boris-spider是一款使用Python语言编写的爬虫框架，于多年的爬虫业务中不断磨合而诞生，相比于scrapy，该框架更易上手，且又满足复杂的需求，支持分布式及批次采集。
Support
Quality
Security
License
Reuse
An open source webapp for scraping: towards a public service for webscraping
Support
Quality
Security
License
Reuse
Rolling Spider software package for Education
Support
Quality
Security
License
Reuse
A web-spider that can run JS based V8 and get AJAX contents, command line mode
Support
Quality
Security
License
Reuse
91porn批量视频、图片下载 ;新手爬虫;novice spider ;多线程
Support
Quality
Security
License
Reuse
NTU CEIBA 資料下載工具
Support
Quality
Security
License
Reuse
House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
Support
Quality
Security
License
Reuse
Some classic web crawler projects.一些经典的爬虫
Support
Quality
Security
License
Reuse
Base on crawler result web path scanner.
Support
Quality
Security
License
Reuse
网易云爬虫解决方案
Support
Quality
Security
License
Reuse
PTT Daily Beauty - 表特日報
Support
Quality
Security
License
Reuse
超高速异步协程Python爬虫
Support
Quality
Security
License
Reuse
Scrape Learning (ctrip)
Support
Quality
Security
License
Reuse
B
Bili23-Downloaderby ScottSloan
下载 Bilibili 视频/番剧/电影/纪录片 等资源
Python 87Updated: 2 y ago License: Permissive (MIT)
87Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
n
newslerby npsolve
A complete automated financial news crawler built on the top of Scrapy framework.
Python 86Updated: 4 y ago License: No License (No License)
86Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
d
dcard-spiderby leVirve
A spider on Dcard. Strong and speedy.
Python 86Updated: 3 y ago License: Permissive (MIT)
86Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
e
es6-crawler-detectby JefferyHus
:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
JavaScript 86Updated: 1 y ago License: Permissive (MIT)
86Updated: 1 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
s
scrapy-tutorialby Wooden-Robot
Scrapy 爬虫框架教程源码
Python 85Updated: 3 y ago License: Permissive (Apache-2.0)
85Updated: 3 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
B
Browser-Default-Stylesby UncaughtTypeError
Search against any element for standardized and default styles from all major rendering engines (WebKit, Blink, Gecko, Trident).
JavaScript 85Updated: 3 y ago License: Permissive (MIT)
85Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
s
spiderby madeindjs
Multithreaded Web spider crawler written in Rust.
Rust 85Updated: 3 y ago License: Permissive (MIT)
85Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
T
Targeted_Literature_Reviews_via_webscrapingby paulamartingonzalez
Web scraping to get articles for a given query. It returns an spreadsheet with titles, abstracts, doi and references of the article
Jupyter Notebook 85Updated: 2 y ago License: No License (No License)
85Updated: 2 y ago License: No License (No License)Support
Quality
Security
License
Reuse
S
SchweizerMesserby monkey-soft
🎯Python 3 网络爬虫实战、数据分析合集 | 当当 | 网易云音乐 | unsplash | 必胜客 | 猫眼 |
HTML 84Updated: 4 y ago License: No License (No License)
84Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
i
irby guilhermecgs
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
Python 83Updated: 3 y ago License: Weak Copyleft (MPL-2.0)
83Updated: 3 y ago License: Weak Copyleft (MPL-2.0)Support
Quality
Security
License
Reuse
t
Support
Quality
Security
License
Reuse
U
Support
Quality
Security
License
Reuse
g
googlesearchby tpeng
Scrape the Google search result with Scrapy.
Python 83Updated: 4 y ago License: No License (No License)
83Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
b
bathyscapheby creekorful
Fast, highly configurable, cloud native dark web crawler.
Go 83Updated: 1 y ago License: Strong Copyleft (GPL-3.0)
83Updated: 1 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
G
GoodreadsScraperby havanagrawal
Scrape data from Goodreads using Scrapy and Selenium :books:
Python 82Updated: 2 y ago License: Permissive (MIT)
82Updated: 2 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
b
bee-universityby beecost
Project thu thập điểm chuẩn đại học 2014 - 2018 và phân tích dữ liệu
Python 82Updated: 4 y ago License: No License (No License)
82Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
s
scrapy_enterprise_architectureby zhengwen09
python scrapy 企业级分布式爬虫开发架构模板
Python 82Updated: 4 y ago License: No License (No License)
82Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
p
podcasts-repoby iRaul
🎙 A collection of podcasts around the web.
JavaScript 82Updated: 3 y ago License: Permissive (MIT)
82Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
Z
ZSpiderby LeoLin9527
Python网络爬虫实战--红薯中文网、企名片、汽车之家、有道翻译、知乎
JavaScript 82Updated: 4 y ago License: No License (No License)
82Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
t
tinydownloaderby abedormancy
a tiny downloader with console panel.
Java 81Updated: 4 y ago License: No License (No License)
81Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
P
ProxyPoolby Greyh4t
Crawl and validate proxies from Internet
Python 81Updated: 4 y ago License: No License (No License)
81Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
R
RastLeakby n4xh4ck5
Tool to automatic leak information using Hacking with engine searches
Python 81Updated: 3 y ago License: Strong Copyleft (GPL-3.0)
81Updated: 3 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
n
news-crawlerby LuChang-CS
A news crawler for BBC News, Reuters and New York Times.
Python 81Updated: 2 y ago License: No License (No License)
81Updated: 2 y ago License: No License (No License)Support
Quality
Security
License
Reuse
w
webbby hardikvasa
Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!
Python 80Updated: 4 y ago License: Permissive (Apache-2.0)
80Updated: 4 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
N
NScrapyby xboxeer
NScrapy is a .net core corss platform Distributed Spider Framework which provide an easy way to write your own Spider
C# 80Updated: 4 y ago License: Permissive (Apache-2.0)
80Updated: 4 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
s
scrapy_helperby facert
Dynamic configurable crawl (动态可配置化爬虫)
CSS 80Updated: 4 y ago License: Permissive (MIT)
80Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
W
Weixinby Python3WebSpider
Sougou Weixin Spider Using Proxy
Python 79Updated: 3 y ago License: No License (No License)
79Updated: 3 y ago License: No License (No License)Support
Quality
Security
License
Reuse
D
DataSpiderby TsingJyujing
A spider library of several data sources.
Python 79Updated: 4 y ago License: Strong Copyleft (GPL-3.0)
79Updated: 4 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
O
Support
Quality
Security
License
Reuse
z
zhihu-scrapyby immzz
A scrapy zhihu crawler
Python 79Updated: 5 y ago License: No License (No License)
79Updated: 5 y ago License: No License (No License)Support
Quality
Security
License
Reuse
h
has_threaded_commentsby aarongough
[DEPRECATED] Add an entire threaded comment system to your Rails application with only 7 lines of code.
Ruby 79Updated: 5 y ago License: Permissive (MIT)
79Updated: 5 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
X
XVideos-PornHub-RedTube-APIby Joel2B
This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own video player.
PHP 79Updated: 1 y ago License: Permissive (MIT)
79Updated: 1 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
g
go-ldap-admin-uiby eryajf
🌉 基于Go+Vue实现的openLDAP后台管理项目
JavaScript 79Updated: 2 y ago License: Strong Copyleft (GPL-3.0)
79Updated: 2 y ago License: Strong Copyleft (GPL-3.0)Support
Quality
Security
License
Reuse
z
zhihu_spiderby Tachone
large-scale user information crawler of zhihu
Python 78Updated: 4 y ago License: No License (No License)
78Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
I
Inventusby nmalcolm
Inventus is a spider designed to find subdomains of a specific domain by crawling it and any subdomains it discovers.
Python 78Updated: 4 y ago License: Permissive (MIT)
78Updated: 4 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
r
random_user_agentby Luqman-Ud-Din
A package to get list of user agents based on filters such as operating system, software name etc..
Python 78Updated: 1 y ago License: Permissive (MIT)
78Updated: 1 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
r
recruitby Frank-qlu
recruit 招聘爬虫+数据分析 1.爬虫： 采用Scrapy 分布式爬虫技术，使用mongodb作为数据存储，爬取的网站Demo为51job，数据我目前爬了有几千条 2.数据处理： 采用pandas对爬取的数据进行清洗和处理 2.数据分析： 采用flask后端获取mongodb数据，前端使用bootstrap3.echarts以及D3的词云图，如果喜欢请star or Fork,预览详见
Python 78Updated: 3 y ago License: Permissive (Apache-2.0)
78Updated: 3 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
b
boris-spiderby Boris-code
boris-spider是一款使用Python语言编写的爬虫框架，于多年的爬虫业务中不断磨合而诞生，相比于scrapy，该框架更易上手，且又满足复杂的需求，支持分布式及批次采集。
Python 78Updated: 3 y ago License: Proprietary (Proprietary)
78Updated: 3 y ago License: Proprietary (Proprietary)Support
Quality
Security
License
Reuse
O
OpenScraperby entrepreneur-interet-general
An open source webapp for scraping: towards a public service for webscraping
Python 78Updated: 3 y ago License: Permissive (MIT)
78Updated: 3 y ago License: Permissive (MIT)Support
Quality
Security
License
Reuse
R
RollingSpiderEduby Parrot-Developers
Rolling Spider software package for Education
C 78Updated: 3 y ago License: No License (No License)
78Updated: 3 y ago License: No License (No License)Support
Quality
Security
License
Reuse
g
grampusSpiderby reichtiger
A web-spider that can run JS based V8 and get AJAX contents, command line mode
C++ 78Updated: 4 y ago License: Permissive (Apache-2.0)
78Updated: 4 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
9
91porn-spiderby look1z
91porn批量视频、图片下载 ;新手爬虫;novice spider ;多线程
Python 77Updated: 3 y ago License: No License (No License)
77Updated: 3 y ago License: No License (No License)Support
Quality
Security
License
Reuse
c
Support
Quality
Security
License
Reuse
a
actor-scraperby apify
House of Apify Scrapers. Generic scraping actors with a simple UI to handle complex web crawling and scraping use cases.
JavaScript 77Updated: 3 y ago License: Permissive (Apache-2.0)
77Updated: 3 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
c
crawler_examplesby liuslnlp
Some classic web crawler projects.一些经典的爬虫
Python 77Updated: 2 y ago License: Permissive (Apache-2.0)
77Updated: 2 y ago License: Permissive (Apache-2.0)Support
Quality
Security
License
Reuse
b
bcrpscanby secfree
Base on crawler result web path scanner.
Python 76Updated: 4 y ago License: No License (No License)
76Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse
c
Support
Quality
Security
License
Reuse
p
ptt-daily-beautyby LarryLuTW
PTT Daily Beauty - 表特日報
Go 76Updated: 3 y ago License: No License (No License)
76Updated: 3 y ago License: No License (No License)Support
Quality
Security
License
Reuse
s
Support
Quality
Security
License
Reuse
c
ctrip_spiderby evanleungc
Scrape Learning (ctrip)
Python 75Updated: 4 y ago License: No License (No License)
75Updated: 4 y ago License: No License (No License)Support
Quality
Security
License
Reuse