cape-api | Our Cape API | REST library
kandi X-RAY | cape-api Summary
kandi X-RAY | cape-api Summary
Our Cape API
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of cape-api
cape-api Key Features
cape-api Examples and Code Snippets
Community Discussions
Trending Discussions on cape-api
QUESTION
I am having some trouble scraping data from the following website: https://www.loft.com.br/apartamentos/sao-paulo-sp?q=pin
When we load the page, it loads the first ~30 posts on real state in the city of Sao Paulo. If we scroll down, it loads more posts.
Usually I would use selenium to get around this - but I want to learn how to do it properly - I imagine that is by fiddling with requests.
By using inspect on chrome, and watching for what happens when we scroll down, I can see a request made which I presume is what retrieves the new posts.
If I copy its content as curl, I get the following command:
...ANSWER
Answered 2021-Feb-05 at 14:301- You did it the proper way! I have been doing it the same way for a long time, and based on my experiences on webscraping, using
your browser network tab
is by far the best way to get info about the requests made on a website, better than any "extension" and/or "plugin" that I know of!!! There is also "burp suit" onkali linux
oron windows
, but again the network tab on the browser is always my number one choice!
2- I have been using the same website that you mentioned!!! It makes my life easier and works seamlessly fine. Of curse, you could do it manually, but the website you mentioned makes it easier and faster for me, and I have been using it for a long time!
3- You could do it manually, it's pretty straightforward, but like I said, the website you mentioned makes it easier and faster .
4- It's probably because you're using
requests.options
, I would try to userequests.get
instead!!!
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install cape-api
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page