rcrawl | Rcrawl is a web crawler written entirely in ruby. It's limited right now by the fact that it will s

rcrawl is a Ruby library. rcrawl has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Remove an absolute URL from the URL Server. Download corresponding document from the internet, grabbing and processing robots.txt first, if available. Feed the document into a rewind input stream(ris) to be read/re-read as needed. Based on MIME type, invoke the process method of the. processing module associated with that MIME type. For example, a link extractor or tag counter module for text/html MIME types, or a gif stats module for image/gif. By default, all text/html MIME types will pass through the link extractor. Each link will be converted to an absolute URL and tested against a (ideally user-supplied) URL filter to determine if it should be downloaded. If the URL passes the filter (currently hard coded as Same Domain?), then call the URL-seen? test. Has the URL been seen before? Namely, is it in the URL Server or has it been downloaded already? If the URL is new, it is added to the URL Server. Back to step 1, repeat until the URL Server is empty.