Crawler Libraries - Page 3

P
Pxerby FoXZilla

JavaScript dotstar image 952 dotVersion:Currentdot
License: Permissive (MIT)

A tool for pixiv.net. 人人可用的P站爬虫

Support
    Quality
      Security
        License
          Reuse
            S

            JavaScript dotstar image 950 dotVersion:Currentdot
            License: Strong Copyleft (GPL-3.0)

            A web crawler/spider which can be used without writing any code with GUI (Service Wrapper)一个可以无代码可视化设计和执行的面向服务架构的爬虫软件

            Support
              Quality
                Security
                  License
                    Reuse
                      c

                      Python dotstar image 949 dotVersion:Currentdot
                      License: Permissive (MIT)

                      Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:

                      Support
                        Quality
                          Security
                            License
                              Reuse
                                l
                                lxSpiderby lixi5338619

                                Python dotstar image 949 dotVersion:Currentdot
                                License: Strong Copyleft (GPL-3.0)

                                爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书、大众点评、推特、脉脉、知乎》

                                Support
                                  Quality
                                    Security
                                      License
                                        Reuse
                                          J
                                          JSpiderby EnjoyScraping

                                          JavaScript dotstar image 934 dotVersion:Currentdot
                                          no licences License: No License (No License)

                                          JSpider会每周更新至少一个网站的JS解密方式,欢迎 Star,交流微信:13298307816

                                          Support
                                            Quality
                                              Security
                                                License
                                                  Reuse
                                                    S
                                                    Spiderby buckyroberts

                                                    Python dotstar image 926 dotVersion:Currentdot
                                                    no licences License: No License (No License)

                                                    Python website crawler.

                                                    Support
                                                      Quality
                                                        Security
                                                          License
                                                            Reuse
                                                              N
                                                              NewPipeExtractorby TeamNewPipe

                                                              Java dotstar image 913 dotVersion:Currentdot
                                                              License: Strong Copyleft (GPL-3.0)

                                                              NewPipe's core library for extracting data from streaming sites

                                                              Support
                                                                Quality
                                                                  Security
                                                                    License
                                                                      Reuse
                                                                        C
                                                                        Camtdby jae-jae

                                                                        JavaScript dotstar image 907 dotVersion:Currentdot
                                                                        no licences License: No License (No License)

                                                                        Chrome multi-threaded download manager extension,based on Aria2 and AriaNg. Chrome多线程下载扩展。

                                                                        Support
                                                                          Quality
                                                                            Security
                                                                              License
                                                                                Reuse
                                                                                  F

                                                                                  Python dotstar image 906 dotVersion:Currentdot
                                                                                  License: Permissive (MIT)

                                                                                  Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

                                                                                  Support
                                                                                    Quality
                                                                                      Security
                                                                                        License
                                                                                          Reuse
                                                                                            z

                                                                                            Java dotstar image 896 dotVersion:Currentdot
                                                                                            License: Proprietary (Proprietary)

                                                                                            zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

                                                                                            Support
                                                                                              Quality
                                                                                                Security
                                                                                                  License
                                                                                                    Reuse
                                                                                                      s
                                                                                                      sukhoiby untwisted

                                                                                                      Python dotstar image 879 dotVersion:Currentdot
                                                                                                      License: Permissive (Apache-2.0)

                                                                                                      Minimalist and powerful Web Crawler.

                                                                                                      Support
                                                                                                        Quality
                                                                                                          Security
                                                                                                            License
                                                                                                              Reuse
                                                                                                                l
                                                                                                                linkedin2usernameby initstring

                                                                                                                Python dotstar image 879 dotVersion:Currentdot
                                                                                                                License: Permissive (MIT)

                                                                                                                OSINT Tool: Generate username lists for companies on LinkedIn

                                                                                                                Support
                                                                                                                  Quality
                                                                                                                    Security
                                                                                                                      License
                                                                                                                        Reuse
                                                                                                                          b
                                                                                                                          bilibili-apiby Nemo2011

                                                                                                                          Python dotstar image 878 dotVersion:Currentdot
                                                                                                                          License: Strong Copyleft (GPL-3.0)

                                                                                                                          哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址:https://github.com/MoyuScript/bilibili-api

                                                                                                                          Support
                                                                                                                            Quality
                                                                                                                              Security
                                                                                                                                License
                                                                                                                                  Reuse
                                                                                                                                    m
                                                                                                                                    magnet-dhtby chenjiandongx

                                                                                                                                    Python dotstar image 869 dotVersion:Currentdot
                                                                                                                                    License: Permissive (MIT)

                                                                                                                                    ✌️ Python3 BitTorrent DHT crawler

                                                                                                                                    Support
                                                                                                                                      Quality
                                                                                                                                        Security
                                                                                                                                          License
                                                                                                                                            Reuse
                                                                                                                                              i
                                                                                                                                              instagram-profilecrawlby timgrossmann

                                                                                                                                              Python dotstar image 849 dotVersion:Currentdot
                                                                                                                                              License: Permissive (MIT)

                                                                                                                                              📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

                                                                                                                                              Support
                                                                                                                                                Quality
                                                                                                                                                  Security
                                                                                                                                                    License
                                                                                                                                                      Reuse
                                                                                                                                                        O
                                                                                                                                                        OnionSearchby megadose

                                                                                                                                                        Python dotstar image 829 dotVersion:Currentdot
                                                                                                                                                        License: Strong Copyleft (GPL-3.0)

                                                                                                                                                        OnionSearch is a script that scrapes urls on different .onion search engines.

                                                                                                                                                        Support
                                                                                                                                                          Quality
                                                                                                                                                            Security
                                                                                                                                                              License
                                                                                                                                                                Reuse
                                                                                                                                                                  t
                                                                                                                                                                  thread-poolby mtrebi

                                                                                                                                                                  C++ dotstar image 820 dotVersion:Currentdot
                                                                                                                                                                  License: Permissive (MIT)

                                                                                                                                                                  Thread pool implementation using c++11 threads

                                                                                                                                                                  Support
                                                                                                                                                                    Quality
                                                                                                                                                                      Security
                                                                                                                                                                        License
                                                                                                                                                                          Reuse
                                                                                                                                                                            n

                                                                                                                                                                            Shell dotstar image 806 dotVersion:Currentdot
                                                                                                                                                                            no licences License: No License (No License)

                                                                                                                                                                            Block bad, possibly even malicious web crawlers (automated bots) using Nginx

                                                                                                                                                                            Support
                                                                                                                                                                              Quality
                                                                                                                                                                                Security
                                                                                                                                                                                  License
                                                                                                                                                                                    Reuse
                                                                                                                                                                                      s
                                                                                                                                                                                      scrapy-seleniumby clemfromspace

                                                                                                                                                                                      Python dotstar image 792 dotVersion:Currentdot
                                                                                                                                                                                      License: Permissive (WTFPL)

                                                                                                                                                                                      Scrapy middleware to handle javascript pages using selenium

                                                                                                                                                                                      Support
                                                                                                                                                                                        Quality
                                                                                                                                                                                          Security
                                                                                                                                                                                            License
                                                                                                                                                                                              Reuse
                                                                                                                                                                                                T

                                                                                                                                                                                                JavaScript dotstar image 775 dotVersion:Currentdot
                                                                                                                                                                                                no licences License: No License (No License)

                                                                                                                                                                                                export thunder lixian url to aria2/wget

                                                                                                                                                                                                Support
                                                                                                                                                                                                  Quality
                                                                                                                                                                                                    Security
                                                                                                                                                                                                      License
                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                          s
                                                                                                                                                                                                          spider_pythonby xingag

                                                                                                                                                                                                          Python dotstar image 774 dotVersion:Currentdot
                                                                                                                                                                                                          License: Permissive (Apache-2.0)

                                                                                                                                                                                                          python爬虫

                                                                                                                                                                                                          Support
                                                                                                                                                                                                            Quality
                                                                                                                                                                                                              Security
                                                                                                                                                                                                                License
                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                    g
                                                                                                                                                                                                                    go-dorkby dwisiswant0

                                                                                                                                                                                                                    Go dotstar image 774 dotVersion:Currentdot
                                                                                                                                                                                                                    License: Permissive (MIT)

                                                                                                                                                                                                                    The fastest dork scanner written in Go.

                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                          License
                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                              c
                                                                                                                                                                                                                              creeperby wspl

                                                                                                                                                                                                                              Go dotstar image 774 dotVersion:Currentdot
                                                                                                                                                                                                                              License: Permissive (Apache-2.0)

                                                                                                                                                                                                                              :paw_prints: Creeper - The Next Generation Crawler Framework (Go)

                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                        h
                                                                                                                                                                                                                                        holiday-cnby NateScarlet

                                                                                                                                                                                                                                        Python dotstar image 772 dotVersion:Currentdot
                                                                                                                                                                                                                                        License: Permissive (MIT)

                                                                                                                                                                                                                                        📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告

                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                  B
                                                                                                                                                                                                                                                  BaiduImageSpiderby kong36088

                                                                                                                                                                                                                                                  Python dotstar image 765 dotVersion:Currentdot
                                                                                                                                                                                                                                                  License: Permissive (MIT)

                                                                                                                                                                                                                                                  一个超级轻量的百度图片爬虫

                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                            f
                                                                                                                                                                                                                                                            fetchbotby PuerkitoBio

                                                                                                                                                                                                                                                            Go dotstar image 760 dotVersion:Currentdot
                                                                                                                                                                                                                                                            License: Permissive (BSD-3-Clause)

                                                                                                                                                                                                                                                            A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                      h
                                                                                                                                                                                                                                                                      house-rentingby kezhenxu94

                                                                                                                                                                                                                                                                      Python dotstar image 756 dotVersion:Currentdot
                                                                                                                                                                                                                                                                      License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                      Possibly the best practice of Scrapy 🕷 and renting a house 🏡

                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                s
                                                                                                                                                                                                                                                                                spidrby postmodern

                                                                                                                                                                                                                                                                                Ruby dotstar image 753 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                License: Permissive (MIT)

                                                                                                                                                                                                                                                                                A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                          i
                                                                                                                                                                                                                                                                                          icrawlerby hellock

                                                                                                                                                                                                                                                                                          Python dotstar image 747 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                          License: Permissive (MIT)

                                                                                                                                                                                                                                                                                          A multi-thread crawler framework with many builtin image crawlers provided.

                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                    B
                                                                                                                                                                                                                                                                                                    BaiduSpiderby BaiduSpider

                                                                                                                                                                                                                                                                                                    Python dotstar image 746 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                    License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                    BaiduSpider,一个爬取百度搜索结果的爬虫,目前支持百度网页搜索,百度图片搜索,百度知道搜索,百度视频搜索,百度资讯搜索,百度文库搜索,百度经验搜索和百度百科搜索。

                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                              z
                                                                                                                                                                                                                                                                                                              zhihu-spiderby MorganZhang100

                                                                                                                                                                                                                                                                                                              Python dotstar image 739 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                              License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                              A web spider for zhihu.com

                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                        C

                                                                                                                                                                                                                                                                                                                        HTML dotstar image 700 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                        License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                                        计算机专业系统性学习资料(python,c,c++,计算机组成,计算机网络,编译原理,电路,谷歌插件,爬虫)

                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                  s
                                                                                                                                                                                                                                                                                                                                  scrapyrtby scrapinghub

                                                                                                                                                                                                                                                                                                                                  Python dotstar image 692 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                  License: Permissive (BSD-3-Clause)

                                                                                                                                                                                                                                                                                                                                  HTTP API for Scrapy spiders

                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                            S
                                                                                                                                                                                                                                                                                                                                            SecCrawlerby Le0nsec

                                                                                                                                                                                                                                                                                                                                            Go dotstar image 691 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                            License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                            一个方便安全研究人员获取每日安全日报的爬虫和推送程序,目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客,持续更新中。

                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                      b
                                                                                                                                                                                                                                                                                                                                                      bililiby SigureMo

                                                                                                                                                                                                                                                                                                                                                      Python dotstar image 688 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                      License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                      :beers: bilibili video (including bangumi) and danmaku downloader | B站视频(含番剧)、弹幕下载器

                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                s
                                                                                                                                                                                                                                                                                                                                                                systemshockby Interrupt

                                                                                                                                                                                                                                                                                                                                                                C dotstar image 677 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                Shockolate - A minimalist and cross platform System Shock source port.

                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                          P
                                                                                                                                                                                                                                                                                                                                                                          PyPttby PyPtt

                                                                                                                                                                                                                                                                                                                                                                          Python dotstar image 674 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                          License: Weak Copyleft (LGPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                          直接連線登入的 PTT library,支援 PTT, PTT2

                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                    x
                                                                                                                                                                                                                                                                                                                                                                                    xeHentaiby fffonion

                                                                                                                                                                                                                                                                                                                                                                                    Python dotstar image 673 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                    License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                                    Doujinshi downloader 绅士漫画下载

                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                              T
                                                                                                                                                                                                                                                                                                                                                                                              TOP250movie_doubanby iphysresearch

                                                                                                                                                                                                                                                                                                                                                                                              Jupyter Notebook dotstar image 673 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                              License: Permissive (BSD-2-Clause)

                                                                                                                                                                                                                                                                                                                                                                                              TOP250豆瓣电影短评:Scrapy 爬虫+数据清理/分析+构建中文文本情感分析模型

                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                        E
                                                                                                                                                                                                                                                                                                                                                                                                        EmailHarvesterby maldevel

                                                                                                                                                                                                                                                                                                                                                                                                        Python dotstar image 670 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                        License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                                                        Email addresses harvester

                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                  P

                                                                                                                                                                                                                                                                                                                                                                                                                  Python dotstar image 661 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                  License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                                                                  Download stuff from PH the easy way.

                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                            x
                                                                                                                                                                                                                                                                                                                                                                                                                            xxl-crawlerby xuxueli

                                                                                                                                                                                                                                                                                                                                                                                                                            Java dotstar image 658 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                            License: Permissive (Apache-2.0)

                                                                                                                                                                                                                                                                                                                                                                                                                            A distributed web crawler framework.(分布式爬虫框架XXL-CRAWLER)

                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                      t
                                                                                                                                                                                                                                                                                                                                                                                                                                      tumblr-utilsby bbolli

                                                                                                                                                                                                                                                                                                                                                                                                                                      Python dotstar image 645 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                      License: Strong Copyleft (GPL-3.0)

                                                                                                                                                                                                                                                                                                                                                                                                                                      Utilities for dealing with Tumblr blogs, Tumblr backup

                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                s
                                                                                                                                                                                                                                                                                                                                                                                                                                                stealerby moyada

                                                                                                                                                                                                                                                                                                                                                                                                                                                Python dotstar image 645 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                                                                                                                                                                抖音、快手、火山、皮皮虾,视频去水印程序

                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                          i
                                                                                                                                                                                                                                                                                                                                                                                                                                                          isbotby omrilotan

                                                                                                                                                                                                                                                                                                                                                                                                                                                          JavaScript dotstar image 645 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                          License: Permissive (Unlicense)

                                                                                                                                                                                                                                                                                                                                                                                                                                                          🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                    t

                                                                                                                                                                                                                                                                                                                                                                                                                                                                    JavaScript dotstar image 642 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                                                                                                                                                                                    tailwindui-crawler downloads the component HTML files locally

                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                              t
                                                                                                                                                                                                                                                                                                                                                                                                                                                                              toolsby ghost123gg

                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Python dotstar image 636 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                                              no licences License: No License (No License)

                                                                                                                                                                                                                                                                                                                                                                                                                                                                              A Python Crawler Framework

                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        s

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Python dotstar image 634 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        use multiple proxies with Scrapy

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  P
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Python-Spiderby lb2281075105

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Python dotstar image 632 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License: Permissive (Apache-2.0)

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            s

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Python dotstar image 631 dotVersion:Currentdot
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License: Permissive (MIT)

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Random User-Agent middleware based on fake-useragent

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      P
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pxerby FoXZilla
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      A tool for pixiv.net. 人人可用的P站爬虫
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      JavaScriptdotstar image 952dotUpdated: 10 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                S
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ServiceWrapper_WebCrawler_GUI_NoCode_Spiderby NaiboWang
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                A web crawler/spider which can be used without writing any code with GUI (Service Wrapper)一个可以无代码可视化设计和执行的面向服务架构的爬虫软件
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                JavaScriptdotstar image 950dotUpdated: 1 y agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          c
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          crawler-user-agentsby monperrus
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 949dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    l
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    lxSpiderby lixi5338619
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书、大众点评、推特、脉脉、知乎》
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Pythondotstar image 949dotUpdated: 9 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              J
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              JSpiderby EnjoyScraping
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              JSpider会每周更新至少一个网站的JS解密方式,欢迎 Star,交流微信:13298307816
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              JavaScriptdotstar image 934dotUpdated: 3 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        S
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Spiderby buckyroberts
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Python website crawler.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Pythondotstar image 926dotUpdated: 1 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  N
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  NewPipeExtractorby TeamNewPipe
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  NewPipe's core library for extracting data from streaming sites
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Javadotstar image 913dotUpdated: 9 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            C
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Camtdby jae-jae
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Chrome multi-threaded download manager extension,based on Aria2 and AriaNg. Chrome多线程下载扩展。
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            JavaScriptdotstar image 907dotUpdated: 1 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      F
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      FunpySpiderSearchEngineby mtianyan
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pythondotstar image 906dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                z
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                zhihu-crawlerby wycm
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Javadotstar image 896dotUpdated: 3 y agodot License: Proprietary (Proprietary)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          sukhoiby untwisted
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Minimalist and powerful Web Crawler.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 879dotUpdated: 3 y agodot License: Permissive (Apache-2.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    l
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    linkedin2usernameby initstring
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    OSINT Tool: Generate username lists for companies on LinkedIn
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Pythondotstar image 879dotUpdated: 10 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              b
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              bilibili-apiby Nemo2011
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址:https://github.com/MoyuScript/bilibili-api
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Pythondotstar image 878dotUpdated: 9 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        m
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        magnet-dhtby chenjiandongx
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ✌️ Python3 BitTorrent DHT crawler
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Pythondotstar image 869dotUpdated: 1 y agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  i
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  instagram-profilecrawlby timgrossmann
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Pythondotstar image 849dotUpdated: 3 y agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            O
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            OnionSearchby megadose
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            OnionSearch is a script that scrapes urls on different .onion search engines.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Pythondotstar image 829dotUpdated: 9 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      t
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      thread-poolby mtrebi
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Thread pool implementation using c++11 threads
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      C++dotstar image 820dotUpdated: 9 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                n
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                nginx-badbot-blockerby mariusv
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Block bad, possibly even malicious web crawlers (automated bots) using Nginx
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Shelldotstar image 806dotUpdated: 3 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          scrapy-seleniumby clemfromspace
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Scrapy middleware to handle javascript pages using selenium
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 792dotUpdated: 1 y agodot License: Permissive (WTFPL)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    T
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ThunderLixianExporterby binux
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    export thunder lixian url to aria2/wget
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    JavaScriptdotstar image 775dotUpdated: 3 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              spider_pythonby xingag
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              python爬虫
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Pythondotstar image 774dotUpdated: 9 mo agodot License: Permissive (Apache-2.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        g
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        go-dorkby dwisiswant0
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        The fastest dork scanner written in Go.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Godotstar image 774dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  c
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  creeperby wspl
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  :paw_prints: Creeper - The Next Generation Crawler Framework (Go)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Godotstar image 774dotUpdated: 1 y agodot License: Permissive (Apache-2.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            h
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            holiday-cnby NateScarlet
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Pythondotstar image 772dotUpdated: 9 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      B
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      BaiduImageSpiderby kong36088
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      一个超级轻量的百度图片爬虫
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pythondotstar image 765dotUpdated: 12 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                f
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                fetchbotby PuerkitoBio
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Godotstar image 760dotUpdated: 3 y agodot License: Permissive (BSD-3-Clause)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          h
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          house-rentingby kezhenxu94
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Possibly the best practice of Scrapy 🕷 and renting a house 🏡
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 756dotUpdated: 3 y agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    spidrby postmodern
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Rubydotstar image 753dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              i
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              icrawlerby hellock
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              A multi-thread crawler framework with many builtin image crawlers provided.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Pythondotstar image 747dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        B
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        BaiduSpiderby BaiduSpider
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        BaiduSpider,一个爬取百度搜索结果的爬虫,目前支持百度网页搜索,百度图片搜索,百度知道搜索,百度视频搜索,百度资讯搜索,百度文库搜索,百度经验搜索和百度百科搜索。
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Pythondotstar image 746dotUpdated: 9 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  z
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  zhihu-spiderby MorganZhang100
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  A web spider for zhihu.com
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Pythondotstar image 739dotUpdated: 3 y agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            C
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ComputerStudentby sfvsfv
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            计算机专业系统性学习资料(python,c,c++,计算机组成,计算机网络,编译原理,电路,谷歌插件,爬虫)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            HTMLdotstar image 700dotUpdated: 10 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      scrapyrtby scrapinghub
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      HTTP API for Scrapy spiders
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pythondotstar image 692dotUpdated: 2 y agodot License: Permissive (BSD-3-Clause)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                S
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                SecCrawlerby Le0nsec
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                一个方便安全研究人员获取每日安全日报的爬虫和推送程序,目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客,持续更新中。
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Godotstar image 691dotUpdated: 11 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          b
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          bililiby SigureMo
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          :beers: bilibili video (including bangumi) and danmaku downloader | B站视频(含番剧)、弹幕下载器
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 688dotUpdated: 2 y agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    systemshockby Interrupt
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Shockolate - A minimalist and cross platform System Shock source port.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Cdotstar image 677dotUpdated: 10 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              P
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              PyPttby PyPtt
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              直接連線登入的 PTT library,支援 PTT, PTT2
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Pythondotstar image 674dotUpdated: 10 mo agodot License: Weak Copyleft (LGPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        x
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        xeHentaiby fffonion
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Doujinshi downloader 绅士漫画下载
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Pythondotstar image 673dotUpdated: 1 y agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  T
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  TOP250movie_doubanby iphysresearch
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  TOP250豆瓣电影短评:Scrapy 爬虫+数据清理/分析+构建中文文本情感分析模型
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Jupyter Notebookdotstar image 673dotUpdated: 9 mo agodot License: Permissive (BSD-2-Clause)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            E
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            EmailHarvesterby maldevel
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Email addresses harvester
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Pythondotstar image 670dotUpdated: 10 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      P
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      PornHub-downloader-pythonby mariosemes
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Download stuff from PH the easy way.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pythondotstar image 661dotUpdated: 10 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                x
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                xxl-crawlerby xuxueli
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                A distributed web crawler framework.(分布式爬虫框架XXL-CRAWLER)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Javadotstar image 658dotUpdated: 10 mo agodot License: Permissive (Apache-2.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          t
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          tumblr-utilsby bbolli
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Utilities for dealing with Tumblr blogs, Tumblr backup
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Pythondotstar image 645dotUpdated: 10 mo agodot License: Strong Copyleft (GPL-3.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    stealerby moyada
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    抖音、快手、火山、皮皮虾,视频去水印程序
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Pythondotstar image 645dotUpdated: 10 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              i
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              isbotby omrilotan
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              JavaScriptdotstar image 645dotUpdated: 10 mo agodot License: Permissive (Unlicense)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        t
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        tailwindui-crawlerby kiliman
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        tailwindui-crawler downloads the component HTML files locally
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        JavaScriptdotstar image 642dotUpdated: 10 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  t
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  toolsby ghost123gg
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  A Python Crawler Framework
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Pythondotstar image 636dotUpdated: 3 y agodotno licences License: No License (No License)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            scrapy-rotating-proxiesby TeamHG-Memex
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            use multiple proxies with Scrapy
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Pythondotstar image 634dotUpdated: 1 y agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      P
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Python-Spiderby lb2281075105
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Pythondotstar image 632dotUpdated: 3 y agodot License: Permissive (Apache-2.0)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              Reuse
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                s
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                scrapy-fake-useragentby alecxe
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Random User-Agent middleware based on fake-useragent
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Pythondotstar image 631dotUpdated: 11 mo agodot License: Permissive (MIT)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                Support
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Quality
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Security
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      License
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Reuse