scrapy-beautifulsoup | Simple Scrapy middleware to process non
kandi X-RAY | scrapy-beautifulsoup Summary
kandi X-RAY | scrapy-beautifulsoup Summary
Simple Scrapy middleware to process non-well-formed HTML with BeautifulSoup
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Initialize middleware .
- Process response body .
- Create an instance from a crawler .
scrapy-beautifulsoup Key Features
scrapy-beautifulsoup Examples and Code Snippets
Community Discussions
Trending Discussions on scrapy-beautifulsoup
QUESTION
Question: Is there a way to integrate BeautifulSoup's html5lib parser into a scrapy project--instead of the scrapy's default lxml parser?
Scrapy's parser fails (for some elements) of my scrape pages.
This only happens every 2 out of 20 pages.
As a fix, I've added BeautifulSoup's parser to the project (which works).
That said, I feel like I'm doubling the work with conditionals and multiple parsers...at a certain point, what's the reason for using Scrapy's parser?
The code does work....it feels like a hack.
I'm no expert--is there a more elegant way to do this?
Much appreciation in advance
Update:
Adding a middleware class to scrapy (from the python package scrapy-beautifulsoup) works like a charm. Apparently, lxml from Scrapy is not as robust as BeautifulSoup's lxml. I didn't have to resort to the html5lib parser--which is 30X+ slower.
ANSWER
Answered 2019-May-08 at 07:31It’s a common feature request for Parsel, Scrapy’s library for XML/HTML scraping.
However, you don’t need to wait for such a feature to be implemented. You can fix the HTML code using BeautifulSoup, and use Parsel on the fixed HTML:
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install scrapy-beautifulsoup
You can use scrapy-beautifulsoup like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page