crawlpy | Python web spider/crawler

crawlpy is a Python library. crawlpy has no bugs, it has no vulnerabilities and it has low support. However crawlpy build file is not available. You can download it from GitHub.

Is the site you want to crawl running on http or https?. The domain or subdomain you want to spider. Nothing outside this domain/subdomain will be touched. 0: Crawl indefinetely until every subpage has been reached.1: Only crawl links on the initial page.2: Crawl links on the initial page and everything found on the links of that page.Note: when you do a login, the login page already counts as one level of depth by scrapy itself, but this is rewritten internally to subtract that depth again, so your output will not show that extra depth. Each array string element is treated as a substring (no regex) and is checked against a FQDN. If any of the specified substrings is found in that URL, it will not be crawled.Note: It does make sense, when you login somewhere, to ignore the logout page, as well as other pages that might delete/disable your current user, so you will not be kicked from your login session during crawl time. By default scrapy ignores pages with status code other than 2xx, so if you know that a 403 page contains actual content with links, just add this here.Note: There is no need to specify 200, as scrapy crawls them by default. true: Do a login prior crawlingfalse: do not login**Note:**When login is set to false, you do not need to fill in the rest of the variables inside the login section. Method required to execute the login. Relative login page (from the base domain, including leading slash) where the post or get will go to. A string that is found on the login page, when the login fails. post or get params. POST or GET params required to login.Examples: username, password, hidden-field-name. true: Login page has a dynamic CSRF token that you want to read out and submit along the normal submit data.false: Login does not require a CSRF token to be submitted.Note: If the login has a static (never-changing) CSRF field, just add the data into the fields sectionNote: Read below about built-in automatic CSRF detection and leave this off at first. The name of the input field which holds the CSRF token. true: Save webpages to diskfalse: Do not save webpages to disk. Absolute or relative path to store webpages to disk.