Python-Tutorials | Python Bootcamp tutorials , with HTML versions | Learning library

First off - scraping 4 million results from a website like Flicker is likely to be unethical. Web scrapers should do their best to respect the website from which they are scraping by minimizing their load on servers. 4 million requests in a short amount of time is likely to get your IP banned. If you used proxies you could get around this but again - highly unethical. You also run into the risk of copyright issues since a lot of the images on flicker are subject to copyright.

If you were to go about doing this you would have to use Scrapy and possibly a Scrapy-Selenium combo. Scrapy is great for running concurrent requests meaning you can request a large number of images at the same time. You can learn more about Scrapy here:https://docs.scrapy.org/en/latest/

The workflow would look something like this:

1. Scrapy makes a request to the website for the html - parse through it to find all tags with class='overlay no-outline'
2. Scrapy makes a request to each url concurrently. This means that the urls won't be followed one by one but instead side by side.
3. As the images are returned they get added to your database/storage space
4. Scrapy (maybe Selenium) scrolls the infinitely scrolling page and repeats without iterating over already checked images (keep index of last scanned item).

This is what Scrapy would entail but I strongly recommend not attempting to scrape 4 million elements. You would probably find that the performance issues you run into would not be worth your time especially since this is supposed to be a learning experience and you will likely never have to scrape that many elements.