linkcrawler | Cross-platform persistent and distributed web crawler link | Crawler library

To fix this I found that you can simply create a global variable and print it.

After you login, you go back to parsing your start url. Scrapy filters out duplicate requests by default, so in your case it stops here. You can avoid this by using 'dont_filter=True' in your request, like this:

What exactly is your use case? Are you primarily interested in the links or content of the pages they lead to? I.e. is there any reason to first store the links in MongoDB and scrape pages later? If you really need to store links in MongoDB, it's best to use an item pipeline to store the items. In the link, there's even example of storing items in MongoDB. If you need something more sophisticated, look at scrapy-mongodb package.

Other than that, there are some comments to the actual code you posted:

• Instead of Selector(response).xpath(...) use just response.xpath(...).
• If you need only the first extracted element from selector, use extract_first() instead of using extract() and indexing.
• Don't use if not not next_page:, use if next_page:.
• The second loop over items is not needed, yield item in the loop over links.