scrapy-spiders | python scripts I have created to crawl various websites | Crawler library

So I have been using selenium to make my scraping. BUT I want to change all the code to Scrapy. The only thing I'm no sure about is that I'm using multiprocessing (python library) to speed up my process. I have researched a lot but I quite don't get it. I have found: Multiprocessing of Scrapy Spiders in Parallel Processes but it doesn't help me because it says that it can be done with Twisted but I haven't found an example yet.

In other forums it says that Scrapy can work with multiprocessing.

Last thing, in scrapy the option CONCURRENT_REQUESTS (settings) has some connection with multiprocessing?

Scrapy 1.4

I am using this script (Run multiple scrapy spiders at once using scrapyd) to schedule multiple spiders at Scrapyd. Before I was using Scrapy 0.19 and was running fine.

I am receiving the error: TypeError: create_crawler() takes exactly 2 arguments (1 given)

So now I dont know if the problem is in Scrapy version or a simple python logical problem (I am new with python)

I did some modifications to check before if the spider is active on the database.

I'm trying to build a system to run a few dozen Scrapy spiders, save the results to S3, and let me know when it finishes. There are several similar questions on StackOverflow (e.g. this one and this other one), but they all seem to use the same recommendation (from the Scrapy docs): set up a CrawlerProcess, add the spiders to it, and hit start().

When I tried this method with all 325 of my spiders, though, it eventually locks up and fails because it attempts to open too many file descriptors on the system that runs it. I've tried a few things that haven't worked.

What is the recommended way to run a large number of spiders with Scrapy?

Edited to add: I understand I can scale up to multiple machines and pay for services to help coordinate (e.g. ScrapingHub), but I'd prefer to run this on one machine using some sort of process pool + queue so that only a small fixed number of spiders are ever running at the same time.

I'm trying to create a function that takes care of a recurring task in multiple spiders. It involves yielding a request that seems to break it. This question is a follow-up from this question.