ureq | A simple , safe HTTP client | HTTP library

In your code you're trying to get the href attribute from the li elements themselves. Actually the li element has a nested p with a nested b which has a nested inside, you need to get that nested a.

Here is a suggestion:

This could be a solution for you:

The issue is that even if it does not find the element, it still prints actual_price which is probably in an outer scope.

You have 2 ways to approach this.

• The 1st is to only print if the element was found, for which you can do:

This might work. I have removed uReq because I prefer using requests ;)

Updated:

What you can do price can be fetch from script tag which reflect in title of the page but it is static not dynamic

Code:

As pointed out by @kmdreko, your code fails to compile because, while ! can be coerced to any T, fn() -> ! cannot be coerced to fn() -> T.

To work around the above, you can declare fail() to return Value, and actually return the "value" of std::process::exit(1). Omitting the semicolon coerces the ! to Value, and you don't have to cheat with a Value::Null:

Welcome to the exciting world of web scraping!

First of all, great job you were on the good track. There are a few points to discuss though.

You essentially have 2 problems here.

1 - How to retrieve the HTML text for all pages (1, ..., 50)?

In web scraping you have mainly to kind of web pages:

1. If you are lucky, the page does not render using javascript and you can use only requests to get the page content
2. You are less lucky, and the page uses JavaScript to render partly or entirely

To get all the pages from 1 to 50, we need to somehow click on the button next at the end of the page. Why? If you check what happens in the network tab from the browser developer, console, you see that a new query getting a JS script to generate the page is fetched for each click to the next button. Unfortunately, we can't render JavaScript using requests

But we have a solution: Headless Browsers (wiki).

In the solution, I use selenium, which is a library that can use a real browser driver (in our case Chrome) to query a page and render JavaScript.

So we first get the web page with selenium, we extract the HTML, we click on next and wait a bit for the page to load, we extract the HTML, ... and so on.

2 - How to extract the text from the PDFs after getting them?

After downloading the PDfs, we can load it into a variable then open it with PyPDF2 and extract the text from all pages. I let you look at the solution code.

Here is a working solution. It will iterate over the first n pages you want and return the text from all the PDF you are interested in:

You'll have to treat the individual table columns differently. For some of them you want just the text, for others you want the hrefs.

Your class filter is not very specific.

The first and second elements are pointing to html nodes which do not contain the link. Thus you are getting error.

A more specific class to check could be: _13oc-S