Python-Tutorial | This repository | Learning library

You can actually do it with a single list comprehension.

Basically, what you have is the right approach, you just need to create a list of lists using your list comprehension.

For each match returned by soup.select, you can extract both the text and href together.

Then, using the csv module, you can pass this list of lists to csv.writerows to create the CSV file for viewing in Excel or other tools, data processing, etc.

You can also optionally prepend a header to the list of lists, if you want, e.g. ['Title', 'URL'].

Here is a full working example:

You can return this formula:

You are converting dictionary to JSON and then again to JSON using JSON.dump twice. Just use it once while writing the file.

First off - scraping 4 million results from a website like Flicker is likely to be unethical. Web scrapers should do their best to respect the website from which they are scraping by minimizing their load on servers. 4 million requests in a short amount of time is likely to get your IP banned. If you used proxies you could get around this but again - highly unethical. You also run into the risk of copyright issues since a lot of the images on flicker are subject to copyright.

If you were to go about doing this you would have to use Scrapy and possibly a Scrapy-Selenium combo. Scrapy is great for running concurrent requests meaning you can request a large number of images at the same time. You can learn more about Scrapy here:https://docs.scrapy.org/en/latest/

The workflow would look something like this:

1. Scrapy makes a request to the website for the html - parse through it to find all tags with class='overlay no-outline'
2. Scrapy makes a request to each url concurrently. This means that the urls won't be followed one by one but instead side by side.
3. As the images are returned they get added to your database/storage space
4. Scrapy (maybe Selenium) scrolls the infinitely scrolling page and repeats without iterating over already checked images (keep index of last scanned item).

This is what Scrapy would entail but I strongly recommend not attempting to scrape 4 million elements. You would probably find that the performance issues you run into would not be worth your time especially since this is supposed to be a learning experience and you will likely never have to scrape that many elements.

Activating a conda environment does not place conda on your PATH. You need to launch the Anaconda Prompt app from your Start menu to get a command-line with conda on your PATH if you didn't check the box to include conda during installation.

Also realize that conda only supports PowerShell as of conda 4.6 which was released in January 2019.

And the Python extension for VS Code works with conda fine. Create a conda environment and the extension will allow you to select it as your environment/interpreter.

Make sure the images are on the same folder as the code.

I know, it's stupid and probably you had already checked that. I had the same exact problem. The answer seems stupid but it worked for me. I assume you have already figured it out. But for the sake of no one repeating it I will leave this here.

From what I have found, if the problem persists you should probably search for "imread not loading images properly" because that's the root of this error.

P.S. I too am following the same tutorial! Keep learning! And for those uninitiated: https://pythonprogramming.net/template-matching-python-opencv-tutorial/

Basically to queue project for training, you have to use below API :

You need to set a symbolic link between calling Python3 and Python since they're effectively the same thing. If you go into a normal command prompt / terminal and type Python3, you may get the same error. Whereas, if you type Python, it will launch the Python interpreter.

Close VSCode

If you're a Windows user you can use mklink to sort it out — open a Command Prompt as an Administrator, then navigate to your Python installation. e.g.:

While writing this question I came up with a solution myself.

In the VS Code launch.json file (which contains the settings for the VS Code Django debugger) contains the following entry, by default:

"args": ["runserver", "--noreload", "--nothreading"]

I changed this to:

"args": ["name_of_management_command"]

Then start the debugger (press f5), and I am debugging my custom management command