dokker | Dokker.js creates professional Javascript code | Runtime Evironment library
kandi X-RAY | dokker Summary
kandi X-RAY | dokker Summary
Dokker is available as npm package. So the easiest way is to install dokker as global module into your project:.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of dokker
dokker Key Features
dokker Examples and Code Snippets
Community Discussions
Trending Discussions on dokker
QUESTION
I am using python scrapy to get the user Reviews comments, which may have multiple pages and I need to click "see more " so as to see more comments.
this is the link to the page I want to crawl: https://en.drivy.com/car-rental/berlin/dacia-dokker-218119
I notice if the review comments more than 10, I need to click "See more" link in order to get the subsequent comments. I also notice the "see more" URL link is https: //en.drivy.com/cars/218119/reviews?page=2&rel=next
However, if I use scrapy to go to the https: //en.drivy.com/cars/218119/reviews?page=2&rel=next, the website redirects me back to https:// en.drivy.com/car-rental/berlin/dacia-dokker-218119 which i cant really get the next ten comments. (i wonder if the website use cookie or session ID and identify my scrapy as new visit)
I know I can use python selenium to open the web page and click "see more" so as to get the comments, however, selenium is very slow and I wish I can use scrapy instead.
Could anyone help me on this? or at least give me a direction to proceed? Thanks in advance.
...ANSWER
Answered 2017-Oct-10 at 07:06You should set "Accept: */*;q=0.5, text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"
header. You'll catch JS object containing texts of comments.
yield Request("https://en.drivy.com/cars/218119/reviews?page=2&rel=next", parse = ..., ...,
headers={'Accept': "*/*;q=0.5, text/javascript, application/javascript, application/ecmascript, application/x-ecmascript"})
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install dokker
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page