scrapers | scrapers for building your own image databases | Scraper library

So I have code that spins up 4 selenium chrome drivers and scrapes data from an element on the web pages. The code can be simplified to something like this:

I've been playing around learning how to create web scrapers using Selenium. One thing I'm struggling with is scraping pages with pagination. I've written a script that i thought would scrape every page

I'm trying to throw together a scrapy spider for a german second-hand products website using code I have successfully deployed on other projects. However this time, I'm running into a TypeError and I can't seem to figure out why.

Comparing to this question ('TypeError: expected string or bytes-like object' while scraping a site) It seems as if the spider is fed a non-string-type URL, but upon checking the the individual chunks of code responsible for generating URLs to scrape, they all seem to spit out strings.

To describe the general functionality of the spider & make it easier to read:

1. The URL generator is responsible for providing the starting URL (first page of search results)
2. The parse_search_pages function is responsible for pulling a list of URLs from the posts on that page.
3. It checks the Dataframe if it was scraped in the past. If not, it will scrape it.
4. The parse_listing function is called on an individual post. It uses the x_path variable to pull all the data. It will then continue to the next page using the CrawlSpider rules.

It's been ~2 years since I've used this code and I'm aware a lot of functionality might have changed. So hopefully you can help me shine a light on what I'm doing wrong?

Cheers, R.

///

The code

I am trying to run a scrapy script with splash, as I want to scrape a javascript based webpage, but with no results. When I execute this script with python command, I get this error: crochet._eventloop.TimeoutError. In addition the print statement in parse method never printed, so I consider something is wrong with SplashRequest. The code that I wrote in order to implement this is that:

I'm trying to scrape some info on tennis matches from a Javascript site using Scrapy and Selenium. The starting URLs are for pages that contain all the matches on a given date. The first task on each page is to make all the matches visible from behind some horizontal tabs - got this covered. The second task is to scrape the match pages that sit behind links that aren't present on the starting URL pages - a specific tag needs to be clicked.

I've found all these tags no problem and have a loop written that uses Selenium to click the tag and yields a Request after each iteration. The issue I'm having is that each time I click through on a link then the page changes and my lovely list of elements detaches itself from the DOM and I get a StaleElementReferenceException error. I understand why this happens but I'm struggling to come up with a solution.

Here's my code so far:

I have been trying to scrape ETF data from iShares.com for an ongoing project for a while now. I am trying to create web scrapers for multiple websites but they are all identical. Essentially I run into two issues:

1. I keep getting the error :"AttributeError: 'NoneType' object has no attribute 'tr'" although I am quite sure that I have chosen the correct table.

2. When I look into the "Inspect elements" on some of the websites, I have to click the "Show more" in order to see the code for all of the rows.

I am not a computer scientist, but I have tried many different approaches which have sadly all been unsuccessful so I hope you can help.

The URL: https://www.ishares.com/uk/individual/en/products/251382/ishares-msci-world-minimum-volatility-ucits-etf

The table can be found on the URL under "Holdings". Alternatively, it can be found under the following paths: JS Path: tbody")> xPath: //*[@id="allHoldingsTable"]/tbody

Code:

I am trying to get my deployment to only deploy replicas to nodes that aren't running rabbitmq (this is working) and also doesn't already have the pod I am deploying (not working).

I can't seem to get this to work. For example, if I have 3 nodes (2 with label of app.kubernetes.io/part-of=rabbitmq) then all 2 replicas get deployed to the remaining node. It is like the deployments aren't taking into account their own pods it creates in determining anti-affinity. My desired state is for it to only deploy 1 pod and the other one should not get scheduled.

I have run into a problem when designing my software.

My software consists of a few classes, Bot, Website, and Scraper.

Bot is the most abstract, executive class responsible for managing the program at a high-level.

Website is a class which contains scraped data from that particular website.

Scraper is a class which may have multiple instances per Website. Each instance is responsible for a different part of a single website.

Scraper has a function scrape_data() which returns the JSON data associated with the Website. I want to pass this data into the Website somehow, but can't find a way since Scraper sits on a lower level of abstraction. Here's the ideas I've tried:

I am currently doing a project which requires me to scrape data from cheapflights. Since it is a group project we have decided to scrape data from the most popular cities on the homepage.

In my case it is a list of destinations with a dictionary of city: url code for city

e.g