nutch-learning | ##修改内容如下 #http

nutch-learning is a Java library. nutch-learning has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However nutch-learning build file is not available. You can download it from GitHub.

##修改内容如下 #http.content.limit 设置为页面内容的大小 nutch-site.xml 从-1修改为5M 510241024 http.content.limit 5242880 The length limit for downloaded content using the file:// protocol, in bytes. If this value is nonnegative (>=0), content longer than it will be truncated; otherwise, no truncation at all. Do not confuse this setting with the http.content.limit setting. #parser.character.encoding.default 设置为正确的编码（针对反爬虫才有效) nutch-default.xml 从windows-1252修改为utf-8 parser.character.encoding.default utf-8 The character encoding to fall back to when no other information is available. #增加是否跳过获取robot的配置 在nutch-site.xml增加如下配置，修改了org.apache.nutch.fetcher.Fetcher.java,org.apache.nutch.parse.ParseSegment.java类 parser.skip.robot true.