webcrawler | Large-scale directional crawler based on scrapy

Answer found from @pcalkins comment

You have two ways to fix this:

Fastest one: Paste your chromedriver.exe file in the same directory that your spider is.

Best one: in SETTINGS.PY put your diver path in SELENIUM_DRIVER_EXECUTABLE_PATH = YOUR PATH HERE

This is you won't use which('chromediver')

Fixed this by rewriting the parse method more carefully. The blog post provided the basic idea: loop over the response body for each keyword you need. But instead of using a for loop, using a list comprehension to build the list of matches worked well with yield

Cloud functions allows you to deploy only your code. The packaging into a container, with buildpack, is performed automatically for you.

If you have already a container, the best solution is to deploy it on Cloud Run. If your webserver listen on the port 5000, don't forget to override this value during the deployment (use --port parameter).

To plug your PubSub topic to your Cloud Run service, you have 2 solutions

• Either manually, you create a PubSub push subscription to your Cloud Run service
• Or you can use EventArc to plug it to your Cloud Run service

In both cases, you need to take care of the security by using a service account with the role run.invoker on the Cloud Run service that you pass to PubSub push subscription or to EventArc

The full URL can be constructed by concatenating a number of variables together.

For example:

The good practice in scraping for the name, Price, links we need to have a good error handling for each of the fields we're scraping. Something like below

I took a bit different approach and I've changed your .csv file to a .txt file as, honestly, whatever you have there doesn't look like CSV structure.

Here's what I came up with:

Problem 1: MongoDriver knows too much about your crawlers. You should separate the driver from crawler1 and crawler2. I'm not sure what your crawl function returns, but I assume it is a list of objects of type A.

You could use an object such as CrawlerService to manage the dependency between MongoDriver and Crawler. This will separate the driver's write responsibility from the crawler's responsibility to crawl. The service will also manage the order of operations which in some cases may be considered good enough.

App Engine requires your main.py file to declare an app variable which corresponds to a WSGI Application.

Since Klein is an asynchronous web framework, it is not compatible with WSGI (which is synchronous).

Your best option would be to use a service like Cloud Run, which would allow you to define your own runtime and use an asynchronous HTTP server compatible with Klein.

You set tables to the first item that is returned by soup.findAll("table", class_='wikitable')[0]. If you take out [0] you write all tables with that class to the tables variable