Web-Scrapper | Web Scrapper using Python and Beautiful Soup | Scraper library

I try to write simple web-scrapper using puppeteer library.

When I get page by url via page.goto, I need to retry if it failed, i.e response code is >= 400.

My snippet:

I'm developing a geolocation web-scrapper with python and selenium. When I enter data in this website, the page refreshes (with the same URL) and when I try to get the data from the latitude and longitude input it prints nothing.

Here's the sample output, it returns an empty string

I did notice that the value tag changes after entering data in

I was recently writing a web-scrapper, and I found myself nesting try/except loops and relying on errors to drive part of my code like the following two segments:

I'm trying to build a web-scrapper to back up my social media accounts (summer project, I know it's useless).

I'm trying to create a nice class structure, so I've come up with the following structure (I accept critique, I'm pretty new to programming):

I am struck up in my below task, hope this idea will be very useful for many. In my company they use several Health Monitoring tool and we do the Health Checks manually. We have a separate health check plan and process, where we go to Web Applications like Site Scope or open certain jars like Spectrum CA (runs as a JNLP file) and check for the server status/health data manually. As we have more than 50 monitoring tools like that I suggested to automate the Health Checkups.

Initially I was suggested to do the automation using Python but I am a JAVA Web developer so I moved to Selenium/Jsoup and use Web-Scrapper concept to fetch the information and do the health checkups. I am able to do considerable progress using Selenium to automating the manual Health Checks using the Web Application.

Problem: As I mentioned we also need to do certain health checkups using jar, desktop apps or jnpl file (applet), I am not sure on how to proceed in automating these. We are not supposed to use any paid tools or unreliable freeware which record and replay the manual desktop operations.

I do not have any idea on those, so can you please suggest me a tool or testing automation tool for these Desktop applications where I should be able to LogIn into it, Enter some text in the TextBox, Search, filter, navigate and get a value and finally verify whether the health is green ?

Note: Few of the application are legacy application so they don't use any api calls.

I run this web-scrapper on my notebook - it uses Firefox (selenium - webdriver) to get the data - it must actually open the Firefox because the data are created by JavaScript. So I wonder if dedicated server can open Firefox and get the data too - I think dedicated servers have no display so it will not work? The script is much more complicated (well 152 lines) - I pasted only the parts which I think will not work. I believe importing the data into PostgreSQL is no problem in dedicated server.