scrapyd-cluster-on-heroku | scalable Scrapyd cluster for distributed web | Crawler library

I did some research on the topic and now I'm confused why you want a proxy for HTTPS.

While it is understandable to want a proxy for HTTP, (HTTP is unencrypted) HTTPS is secure.

Could it be possible your proxy is not connecting because you don't need one?

I am not a proxy expert, so I apologize if I'm putting out something completely stupid.

I don't want to leave you completely empty-handed though. If you are looking for complete privacy, I would suggest a VPN. Both Windscribe and RiseUpVPN are free and encrypt all your data on your computer. (The desktop version, not the browser extension.)

While this is not a fully automated process, it is still very effective.

It seems that you were missing a VPC Endpoint for DynamoDB, since your Glue Jobs run in a private VPC when you write to Redshift.

Taking a stab at an answer here with no experience of the libraries.

It looks like Scrapy Crawlers themselves are single-threaded. To get multi-threaded behavior you need to configure your application differently or write code that makes it behave mulit-threaded. It sounds like you've already tried this so this is probably not news to you but make sure you have configured the CONCURRENT_REQUESTS and REACTOR_THREADPOOL_MAXSIZE.

https://docs.scrapy.org/en/latest/topics/settings.html?highlight=thread#reactor-threadpool-maxsize

I can't imagine there is much CPU work going on in the crawling process so i doubt it's a GIL issue.

Excluding GIL as an option there are two possibilities here:

1. Your crawler is not actually multi-threaded. This may be because you are missing some setup or configuration that would make it so. i.e. You may have set the env variables correctly but your crawler is written in a way that is processing requests for urls synchronously instead of submitting them to a queue.

To test this, create a global object and store a counter on it. Each time your crawler starts a request increment the counter. Each time your crawler finishes a request, decrement the counter. Then run a thread that prints the counter every second. If your counter value is always 1, then you are still running synchronously.

The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.

Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):

I am trying to change setting from command line while starting a scrapy crawler (Python 3.7). Therefore I am adding a init method...
...
scrapy crawl test -a delay=5

1. According to scrapy docs. (Settings/Command line options section) it is requred to use -s parameter to update setting
   scrapy crawl test -s DOWNLOAD_DELAY=5

2. It is not possible to update settings during runtime in spider code from init or other methods (details in related discussion on github Update spider settings during runtime #4196

I think that there's no way to use chrome/chromium on m1 docker.

• no binary for chrome arm64 linux
• when running chrome on amd64 container with m1 host crashes - docker docs
• chromium could be installed using snap, but snap service not running on docker (without snap, having 127 error because binary from apt is empty) - issue report

Chromium supports arm ubuntu; I tried using chromium instead of chrome.

But chromedriver officially does not support arm64; I used unofficial binary on electron release. https://stackoverflow.com/a/57586200/11853111

Finally, I've decided to use gechodriver and firefox while using docker.

It seamlessly works regardless of host/container architecture.

Use a here-document

This works,

The Glue crawler is only used to identify the schema that your data is in. Your data sits somewhere (e.g. S3) and the crawler identifies the schema by going through a percentage of your files.

You then can use a query engine like Athena (managed, serverless Apache Presto) to query the data, since it already has a schema.

If you want to process / clean / aggregate the data, you can use Glue Jobs, which is basically managed serverless Spark.