Crawler-Detect | 🕷 CrawlerDetect is a PHP class | Crawler library

by JayBizzle PHP Version: v1.2.114 License: MIT

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | Crawler-Detect Summary

Crawler-Detect is a PHP library typically used in Automation, Crawler applications. Crawler-Detect has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers.

Support

Quality

Security

License

Reuse

Support

Crawler-Detect has a medium active ecosystem.

It has 1779 star(s) with 232 fork(s). There are 53 watchers for this library.

It had no major release in the last 12 months.

There are 5 open issues and 134 have been closed. On average issues are closed in 152 days. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of Crawler-Detect is v1.2.114

Quality

Crawler-Detect has 0 bugs and 0 code smells.

Security

Crawler-Detect has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

Crawler-Detect code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

Crawler-Detect is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

Crawler-Detect releases are available to install and integrate.

Installation instructions are not available. Examples and code snippets are available.

Crawler-Detect saves you 655 person hours of effort in developing the same functionality from scratch.

It has 1520 lines of code, 10 functions and 6 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed Crawler-Detect and discovered the below as its top functions. This is intended to give you an instant insight into Crawler-Detect implemented functionality, and help decide if they suit your requirements.

Set the HTTP headers .
Determine if the request is crawler .
Set the user agent
Compile a regular expression .
Get the HTTP headers .
Get all data .

Get all kandi verified functions for this library.

Crawler-Detect Key Features

No Key Features are available at this moment for Crawler-Detect.

Crawler-Detect Examples and Code Snippets

No Code Snippets are available at this moment for Crawler-Detect.

Community Discussions

Trending Discussions on Crawler-Detect

HTML 5 Storage against Crawler and Bots

QUESTION

HTML 5 Storage against Crawler and Bots

Asked 2018-Jan-10 at 12:52

I'm currently coding a view / visitor counter, which checks if the visitor is a human or a bot / crawler. I already found some solutions, which I use. One of these is a cookie (which is set with Javascript), but some Bots already allows to set cookies, and some humans doesn't. :/

Now I'm questioning if it's maybe more effective to set something in the HTML 5 storage. It's maybe more unlikely that bots supports the HTML 5 storage technology as cookies, or? And is it possible to disable the HTML 5 storage on any (human) browser? I haven't found any option yet.

Thanks, Sam.

PS.: Some useful / informative stuff I already know / use within my script:

...

ANSWER

Answered 2018-Jan-10 at 12:52

Crawlers don't normally execute Javascript. So you can monitor visits with JS. You can also filter out the user agents of bots.

Here is the first list I found with crawler user agents.

https://deviceatlas.com/blog/list-of-web-crawlers-user-agents

Source https://stackoverflow.com/questions/48186860

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install Crawler-Detect

You can download it from GitHub.
PHP requires the Visual C runtime (CRT). The Microsoft Visual C++ Redistributable for Visual Studio 2019 is suitable for all these PHP versions, see visualstudio.microsoft.com. You MUST download the x86 CRT for PHP x86 builds and the x64 CRT for PHP x64 builds. The CRT installer supports the /quiet and /norestart command-line switches, so you can also script it.

Support

If you find a bot/spider/crawler user agent that CrawlerDetect fails to detect, please submit a pull request with the regex pattern added to the $data array in Fixtures/Crawlers.php and add the failing user agent to tests/crawlers.txt. Failing that, just create an issue with the user agent you have found, and we'll take it from there :).

Find more information at: