scraper | Node.js based scraper using headless chrome | Scraper library

I'm doing some scraping, but as I'm parsing approximately 4000 URL's, the website eventually detects my IP and blocks me every 20 iterations.

I've written a bunch of Sys.sleep(5) and a tryCatch so I'm not blocked too soon.

I use a VPN but I have to manually disconnect and reconnect it every now and then to change my IP. That's not a suitable solution with such a scraper supposed to run all night long.

I think rotating a proxy should do the job.

Here's my current code (a part of it at least) :

I am having issues with my eBAY Scraper and can not work out why. Although it is pulling the data off fine, it misses SOME of the data OFF for the first row and then for each first row of every Loop and therefore the data is not in the correct row.

Q) Why is it missing the data at the start and then for each loop?

I think It may have something to do with the title extracting slower that the rest of the items, however I can not work it out as I am very limited with vba. I have attached a demo, for your viewing.

I am not looking for a full rewite of the code, just pointing in the right direction or a SLIGHT change to MY code. As I stated I and very limited in vba, I can understand my code, anything more advanced will be out of my depth.

Demo Download - Download Excel File

WebSite - Ebay.co.uk

Ebay Product Page - Prodcts Shown may vary browser to browser

I have colour coded it so you can see better

This is what it is doing

When It Should be This

For some reason it misses out Price, Condition, Former Price & Discount for the first item on start and EVERY Loop. For every loop that it misses the items out the Price, Condition, Former Price & Discount become MORE out of line

1st Loop - Items are NOW 2 rows out of line

2nd Loop - Items are NOW 3 rows out of line

As I searched 3 pages (2 pages + 1 extra) and it looped 3 time it has missed the first row on each loop. I am 3 rows out. I think this may have too do with the Title of the item as it extracts a bit slower then the rest of the items

End Of Extraction

This is my code

I'm trying to store some fields derived from a webpage in mysql table. The script that I've created can parse the data and store them in the table. However, as the username is non-english, the table stores the name as ????????? ????????? instead of Αθανάσιος Σουλιώτης.

Script I've tried with:

I have been working on prometheus and Python where I want to be able to have multiple scripts that writes to Promethethus.

Currently I have done 2 scripts: sydsvenskan.py

Currently I am using this...

i am trying to write a scraper but i have faced with an issue. I can parse "class in spans" and "class in div" but when i try to parse "id in span" it doesn't print the data i want.

I am new to Selenium, Python, and programming in general but I am trying to write a small web scraper. I have encountered a website that has multiple links but their HTML code is not available for me using

I am trying to make a reddit scraper. It works fine however I get issues when there are emojis. To try and fix this I found this function on another question.

I have JSON data with ISO date, and I want to get all the data that "date_created" is within the date range, regardless of what the time is, and without modifying the value of the JSON data.

date range sample: start date: 2021-05-25T16:00:00.000Z, end date: 2021-05-28T16:00:00.000Z

sample of JSON data: