Spiders | Learn the web crawler | Crawler library

I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.

Solution: If you have the same problem,

1. Find the package

I looked at that post

1. Add it to sys.path (which will also add it to PYTHONPATH)

Add this code at the top of your script (in my case, the pipelines.py):

Your code will run 2 spiders simultaneously.
Running spiders sequentially (start Zoopy2 after completion of Zoopy1) can be achieved with @defer.inlineCallbacks:

For those two, you can use the following css selector:

The key line in the log is this one

I ditched the Crawl Spider and converted to a regular spider, and things are working fine now.

Thanks to Serhii's suggestion, I found that the issue was due to "accept-encoding": "gzip, deflate, br": I accepted compressed sites but did not handle them in scrapy.

Adding scrapy.downloadermiddlewares.httpcompression or simply removing the accept-encoding line fixes the issue.

I have implemented a similar method in my web scraping module.

Below is the implementation you can look at and take reference from.

please remove the below line from your code

To run a containerised app, it must be installed in a container image first - you don't want to install any software on the host machine.

For linux there are off-the-shelf container images for everything which is probably what your docker desktop environment was using; I see 1051 results on docker hub search for scrapy but none of them are windows containers.

The full process of creating a windows container from scratch for an app is:

• Get steps to manually install the app (scrapy and its dependencies) on Windows Server - ideally test in a virtualised environment so you can reset it cleanly
• Convert all steps to a fully automatic powershell script (e.g. for conda, need to download the installer via wget, execute the installer etc.
• Optionaly, test the powershell steps in an interactive container
  • docker run -it --isolation=process mcr.microsoft.com/windows/servercore:ltsc2019 powershell
  • This runs a windows container and gives you a shell to verify that your install script works
  • When you exit the shell the container is stopped
• Create a Dockerfile
  • Use mcr.microsoft.com/windows/servercore:ltsc2019 as the base image via FROM
  • Use the RUN command for each line of your powershell script

I tried installing scrapy on an existing windows Dockerfile that used conda / python 3.6, it threw error SettingsFrame has no attribute 'ENABLE_CONNECT_PROTOCOL' at a similar stage.

However I tried again with miniconda and python 3.8, and was able to get scrapy running, here's the dockerfile: