robotstxt | robots.txt file parsing and checking for R | Sitemap library

Your code is working fine as your expectation and the problem was in pagination portion and I've made the pagination in start_urls which type of pagination is always accurate and more than two times faster than if next page.

I think your error is that you are trying to parse instead of starting the requests.

Change:

I run your code on a clean, virtual environment and it is working as intended. It doesn't give me a KeyError either but has some problems on various xpath paths. I'm not quite sure what you mean by filtering out too much data but your code hands me this output:

You can fix the text errors (on product category, part number and description) by changing xpath variables like this:

Where several user agents are recognized in the robots.txt file, Google will follow the most specific.

If you want all of Google to be able to crawl your pages, you don't need a robots.txt file at all.

If you want to block or allow all of Google's crawlers from accessing some of your content, you can do this by specifying Googlebot as the user agent.

To select element inside element you have to put a dot . in front of the XPath expression saying "from here".
Otherwise it will bring you the first match of (//tr/td[@class='time']/span)[1]/text() on the entire page each time, as you see.
Also, since you are iterating per each row it should be row.xpath..., not rows.xpath since rows is a list of elements while each row is an element.
Also, to apply search on a web element according to XPath locator you should use find_element_by_xpath method, not xpath.

Note: As I do not have 50 reputation to comment that's why I am answering here.

The problem is in function naming, your function should be def start_requests(self) instead of def start_request(self).

The first requests to perform are obtained by calling the start_requests() method which (by default) generates Request for the URLs. But, in your case it never gets into that function due to which the requests are never made for those URLs.

Your code after small change

I found a solution to my problem. In fact, just as I suspected, there was a missing directory to my PYTHONPATH. It was the directory that contained the gtts package.

Solution: If you have the same problem,

1. Find the package

I looked at that post

1. Add it to sys.path (which will also add it to PYTHONPATH)

Add this code at the top of your script (in my case, the pipelines.py):

The key line in the log is this one

I have implemented a similar method in my web scraping module.

Below is the implementation you can look at and take reference from.