webarticle2text | main article text from an arbitrary webpage

webarticle2text is a HTML library. webarticle2text has no bugs, it has no vulnerabilities, it has a Weak Copyleft License and it has low support. You can download it from GitHub.

This project is obsolete and now only serves as a reference. I recommend you instead use newspaper, which is an order-of-magnitude more accurate than any other article extraction library I've encountered. Please see compare.csv for a performance comparison of several similar tools. This attempts to locate and extract the largest cluster of text in a webpage. It does this by walking the DOM-tree, identifying all text segments and their depth inside the DOM, appends all text at roughly the same depth, and then returns the chunk with the largest total length. This approach usually works well with typical news sites where one news article is displayed per URL. This approach usually fails with URLs displaying multiple news blurbs (e.g. news aggregators).