scrapy | ScrapySeleniumDjango政府网站爬虫 | Crawler library

From scrapy’s documentation Passing additional data to callback functions, you basically want to pass the code to the data callback in Request’s cb_kwargs argument,

To get all codes, you could iterate on (COL-A, COL-B) pairs, not simply on COL-A values. Here we return the 2d numpy array, thus the list of rows, where each row is the COL-A, COL-B pair:

This is a and has to be in which should have action="URL"

You have this url in

You use too many elements in css - you should use something simpler because some elements may exists in browser's DOM tree (which it shows in DevTools) but not in real HTML (which you get from server). ie. tbody usually doesn't exist in HTML

This gives me results

With @url http://www.amazon.com/s?field-keywords=selfish+gene I get also error 503.

Probably it is very old example - it uses http but modern pages use https - and amazone could rebuild page and now it has better system to detect spamers/hackers/bots and block them.

If I use @url http://toscrape.com/ then I don't get error 503 but I still get other error FAILED because it needs some code in parse()

@scrapes Title Author Year Price means it has to return item with keys Title Author Year Price

this XPath:

'normalize-space(//div[@id="Content"]//div[@id="programDetails"]//div[@id="selfReportedProgramDetails"]//div[@id="hoursOfOperation"]//span[@class="hoursItem"]//span[@class="times"]/text())'):

will give only one result because of the normalize-space() function with all whitespace collapsed.

So to get the actual text-nodes for those spans remove the normalize-space around your XPath.

The second XPath starts with double slash, meaning, it will search from the root all nodes. To search from current context use the .

for more info on // vs .// see this good answer

I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.

Solution: If you have the same problem,

1. Find the package

I looked at that post

1. Add it to sys.path (which will also add it to PYTHONPATH)

Add this code at the top of your script (in my case, the pipelines.py):

You need to declare variable inside class, if you want to initialize do that in constructor.

In case if pipeline code, spider code and process launcher stored in the same file
You can use __main__ in path to enable pipeline:

Not sure about usage of Item classes here for this.. nested items.
The fastest way to achieve this is to yield dictionaries (not Item class objects):