parse_urls | parse_urls 解析任何格式的URL

My Scrapy code doesn't work. I'm trying to do scraping of the forum but receiving an error. Here is my code:

I want to iterate through all the category urls and scrap the content from each page. Although urls = [response.xpath('//ul[@class="flexboxesmain categorieslist"]/li/a/@href').extract()[0]] in this code I have tried to fetch only the first category url but my goal is to fetch all urls and the content inside each urls.

I'm using scrapy_selenium library. Selenium page source is not passing to the 'scrap_it' function. Please review my code and let me know if there's anything wrong in it. I'm new to scrapy framework.

Below is my spider code -

I keep getting error AttributeError: __aexit__ on the code below, but I don't really understand why this happens.

My Python version is: 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]

I'm scraping a site that contains a couple dozen base urls that ultimately link to several thousand xml pages that I parse, turn into a Pandas dataframe, and eventually save to a SQLite database. I multiprocess the download/parsing stages to save time, but the script silently hangs (stops collecting pages or parsing XML) after a certain number of pages (not sure how many; between 100 and 200).

Using the same parser but doing everything sequentially (no multiprocessing) doesn't give any problems, so I suspect I'm doing something wrong with the multiprocessing. Perhaps creating too many instances of the Parse_url class and clogging memory?

Here's an overview of the process: