scrapers | Lots and lots of web scrapers | Scraper library

Ok I understand what you're looking for. And sorry to say you're out of luck. At least as far as my knowledge of python goes. You can do it two ways.

1. Use importlib to search through a folder/package tha contains those files and imports them into a list or dict to be retrieved. However you said you wanted to avoid this but either way you would have to use importlib. And #2 is the reason why.

2. Use a Base class that when inherited it's __init__ call adds the Derived class to a list or object that stores it and you can retrieve it via a class object. However the issue here is that if you move your derived class into a new file, that code wont run until you import it. So you would still need to explicitly import the file or implicitly import it via importlib (dynamic import).

So you'll have to use importlib (dynamic import) either way.

You have several problems in your code:

• you need to wait for the item to be available on the page. looks like the priceblock_ourprice is generated after the page is send to the client.

  In puppeteer, there's a build in function to wait for a certain selector:

Just tell yrou Scraper class that this attribut exists

Problem is that k.domain returns bbc and you wrote url = 'bbc.co.uk' so one these solutions

• use url = 'bbc.co.uk' along with k.registered_domain
• use url = 'bbc' along with k.domain

And add a parameter in the scrape method to get the response

Do as you did in __init_subclass__ or use cls.scrapers.

Try replacing this:

once mount is complete drive.mount('/content/gdrive') file can be accessed like

You can try to use python threading.

Learn more here

It basically allows you to run it on different threads

example:

To run a containerised app, it must be installed in a container image first - you don't want to install any software on the host machine.

For linux there are off-the-shelf container images for everything which is probably what your docker desktop environment was using; I see 1051 results on docker hub search for scrapy but none of them are windows containers.

The full process of creating a windows container from scratch for an app is:

• Get steps to manually install the app (scrapy and its dependencies) on Windows Server - ideally test in a virtualised environment so you can reset it cleanly
• Convert all steps to a fully automatic powershell script (e.g. for conda, need to download the installer via wget, execute the installer etc.
• Optionaly, test the powershell steps in an interactive container
  • docker run -it --isolation=process mcr.microsoft.com/windows/servercore:ltsc2019 powershell
  • This runs a windows container and gives you a shell to verify that your install script works
  • When you exit the shell the container is stopped
• Create a Dockerfile
  • Use mcr.microsoft.com/windows/servercore:ltsc2019 as the base image via FROM
  • Use the RUN command for each line of your powershell script

I tried installing scrapy on an existing windows Dockerfile that used conda / python 3.6, it threw error SettingsFrame has no attribute 'ENABLE_CONNECT_PROTOCOL' at a similar stage.

However I tried again with miniconda and python 3.8, and was able to get scrapy running, here's the dockerfile: