web-scraper | Perl web scraping toolkit | Scraper library

I just started learning about web scraping and I found this tutorial: https://www.mundojs.com.br/2020/05/25/criando-um-web-scraper-com-nodejs/

It works fine, however I'm trying to get different elements from the same webpage: https://ge.globo.com/futebol/brasileirao-serie-a/

With the group of classes of the tutorial it brings all the elements with the selected class, but with other classes it doesn't work:

As can be seen all fifty elements with the class ranking-item-wrapper are returned, but if I select elements with the class lista-jogos__jogo it doesn't return anything:

I don't get why I'm getting this error, since I'm doing exectly the same thing as it is done in the tutorial.

Here is a short version of the code:

I've been working on a web-scraper to scrape the CoinEx website so I can have the live trades of Bitcoin in my program. I scraped this link and I was expecting to get all the information related to the class_="ticker-item" but the return was "--". I think it's something with the scraping policy but is there a way I can bypass this. Like to mimic whatever a regular browser has. I also tried using headers but the result was the same. My Code :

I'm having problem with print page numbers for my web-scraper.

Here I have page range

I'm having a bit of trouble with my wikipedia table web-scraper: The trouble is that it will not read the text in the cells. I have defined the table - no problems there, i have defined the rows, no problem there. My code looks like this:

I am working to write a web-scraper for a school project which catalogs all valid URLs found on the page, and can follow a URL to the next webpage and perform the same action; up to a set number of layers.

quick code intent:

1. function takes a BeautifulSoup type, a url (to indicate where it started), the layer count, and the maximum layer depth
2. check page for all href lines
3. append to list of 'results' is populated each time an href tag is found containing a valid url (starts with http, https, HTTP, HTTPS; which I know may not be the perfect way to check but for now its what I am working with)
4. the layer is incremented by 1 each time a valid URL is found the recursiveLinkSearch() function called again
5. when layer count is reached, or no href's remain, return results list

I am very out of practice with recursion, and am hitting an issue with python adding a 'None' to the list "results" at the end of the recursion.

This link [https://stackoverflow.com/questions/61691657/python-recursive-function-returns-none] indicates that it may be where I am exiting my function from. I am also not sure I have recursion operating properly because of the nested for loop.

Any help or insight on recursion exit strategy is greatly appreciated.

I'm building a simple web-scraper (scraping jobs from indeed.com) for practice and I'm trying to implement the following method (low_salary?(salary)). The aim is for the method to compare a minimum (i.e. desired) salary, compare it with the offered salary contained in the job object (@salary):

I'm trying to store some fields derived from a webpage in mysql table. The script that I've created can parse the data and store them in the table. However, as the username is non-english, the table stores the name as ????????? ????????? instead of Αθανάσιος Σουλιώτης.

Script I've tried with:

I just switched from C to Python and to get some practice I want to code a simple web-scraper for price comparison. It works so far that the program goes to every website I tell it to, and it gives me the information of the website back as html. But when I try to tell BeautifulSoup to just find the prices, the output is 'None'. So I think that the html address, I am passing to BeautifulSoup as the price information is wrong.

I would be really grateful if anyone could help me with that problem or just has some tips and tricks for a beginner! I will add my python code and the link to the website since it looks kinda messy if I put the html code here, just tell me if you need anything more. Thank you!

https://www.momox.de/offer/9783833879500

I would just need the part where it says 8,87€ (or whatever €, price is changing constantly), but looks like i got the wrong part of the html code..

I have a dropdown and lets say the dropdown's code looks something like this: