jsonlines | python library to simplify

CrawlSpider is meant for cases where you want to automatically follow links that match a particular pattern. If you want to obtain information from previous pages, you have to parse each page individually and pass information around via the meta request argument or the cb_kwargs argument. You can add any information to the meta value in any of the parse methods.

I have refactored the code above to use the normal scrapy Spider class and have passed the country value from the first page in the meta keyword and then captured it in subsequent parse methods.

Your use case is suited for using scrapy crawl spider. You can write rules on how to extract links to the properties and how to extract links to the next pages. I have changed your code to use a crawl spider class and I have changed your FEEDS settings to use the recommended settings. FEED_URI and FEED_FORMAT are deprecated in newer versions of scrapy.

Read more about the crawl spider from the docs

From the code snippet you have provided, your scraper is set up efficiently as it is yielding many requests at a go which lets scrapy handle the concurrency.

There are a couple of settings you can tweak to increase the speed of scraping. However, note that the first rule of scraping is that you should not harm the website you are scraping. See below sample of the settings you can tweak.

1. Increase the value of CONCURRENT_REQUESTS. Defaults to 16 in scrapy
2. Increase the value of CONCURRENT_REQUESTS_PER_DOMAIN. Defaults to 8 in scrapy
3. Increase Twisted IO thread pool maximum size so that DNS resolution is faster REACTOR_THREADPOOL_MAXSIZE
4. Reduce log level LOG_LEVEL = 'INFO'
5. Disable cookies if you do not require them COOKIES_ENABLED = False
6. Reduce download timeout DOWNLOAD_TIMEOUT = 15
7. Reduce the value of DOWNLOAD_DELAY if your internet speed is fast and you are sure the website you are targeting is fast enough. This is not recommended

Read more about these settings from the docs

If the above settings do not solve your problem, then you may need to look into distributed crawling

Your code has 2 issues. One your xpath selectors are not correct and two you are not using scrapy playwright therefore the clicks are not being done. Looping and changing the item index is not correct because once you click an item, that item is removed from the DOM and therefore the next item is now at the first index. Also, to enable scrapy-playwright you need to have at least these additional settings:

Your code has quite a lot of redundant snippets. I have removed the redundant code and also implemented your request to have all the lyrics captured. Also all the information is available on the lyrics page so there's no need to pass the loader item around. You can simply crawl all the information from the lyrics page.

spider.py:

You also need the headers in response.follow

If the dict never have more than one item, you can do this:

The problem was on this line:

file = openai.File.create(file=open(jsonFileName), purpose="search")

It returns the call with a file ID and status uploaded which makes it seem like the upload and file processing is complete. I then passed that fileID to the search API, but in reality it had not completed processing and so the search API threw the error openai.error.InvalidRequestError: File is still processing. Check back later.

The returned file object looks like this (misleading):

It worked in google colab because the openai.File.create call and the search call were in 2 different cells, which gave it the time to finish processing as I executed the cells one by one. If I write all of the same code in one cell, it gave me the same error there.

So, I had to introduce a wait time for 4-7 seconds depending on the size of your data, time.sleep(5) after openai.File.create call before calling the openai.Engine("davinci").search call and that solved the issue. :)