muffet | A perl/moose spider. | Regex library

muffet is a Perl library typically used in Utilities, Regex applications. muffet has no bugs, it has no vulnerabilities, it has a Strong Copyleft License and it has low support. You can download it from GitHub.

muffet is a web spider written in perl and moose. the problem that i was trying to solve was to spider newint.org to make a xapian index file, so its targetted at that usage. however, it can spit out xml for a google sitemap or raw text for debugging as well. bear in mind it doesn't respect robots.txt. however you can use a xpath_noindex in pages you want nofollowed. you can also specify the skip_urls parameter, which does a regex match and skips matching urls. i don't have the time to offer any kind of support, but i occasionally fix bugs or add features. usage: muffet.pl [-?] [long options...] -? --usage --help prints this usage information. --user user to attempt http auth with --pass password to attempt http auth with --format output in this format currently xapiani, sitemap or raw --xpath_body where to look for our body data as an xpath expression --xpath_sample where to look for our summary data as an xpath expression --xpath_category where to look for our title as an xpath expression --xpath_tags where to look for our tags as an xpath expression --xpath_title where to look for our title in html docs as an xpath expression --xpath_modified where to look for our modification time as an xpath expression --xpath_noindex where to look for our noindex elements as an xpath expression --xap_db_file database file for xapian output --xap_tmp_file temp file for xapian output defaults to