crawl | A simple sitemap scrapper written in Go | Sitemap library

Try the following approach:

You can make this dynamic by finding the headings first then using sibling selectors to find the images

useEffect is a React hook and we put at the bottom of the code to take priority at this block of the code. It is basically a hook replacement for the "old-school" lifecycle methods componentDidMount, componentDidUpdate and componentWillUnmount. It allows you to execute lifecycle tasks without a need for a class component.

You have two issues with your code. First, you have two Rules in your crawl spider and you have included the deny restriction in the second rule which never gets checked because the first Rule follows all links and then calls the callback. Therefore the first rule is checked first and therefore it does not exclude the urls you don't want to crawl. Second issue is that in your second rule, you have included the literal string of what you want to avoid scraping but deny expects regular expressions.

Solution is to remove the first rule and slightly change the deny argument by escaping special regex characters in the url such as -. See below sample.

Rootcause has been identified as submitter property wasn't created with flag refinable

Taking a stab at an answer here with no experience of the libraries.

It looks like Scrapy Crawlers themselves are single-threaded. To get multi-threaded behavior you need to configure your application differently or write code that makes it behave mulit-threaded. It sounds like you've already tried this so this is probably not news to you but make sure you have configured the CONCURRENT_REQUESTS and REACTOR_THREADPOOL_MAXSIZE.

https://docs.scrapy.org/en/latest/topics/settings.html?highlight=thread#reactor-threadpool-maxsize

I can't imagine there is much CPU work going on in the crawling process so i doubt it's a GIL issue.

Excluding GIL as an option there are two possibilities here:

1. Your crawler is not actually multi-threaded. This may be because you are missing some setup or configuration that would make it so. i.e. You may have set the env variables correctly but your crawler is written in a way that is processing requests for urls synchronously instead of submitting them to a queue.

To test this, create a global object and store a counter on it. Each time your crawler starts a request increment the counter. Each time your crawler finishes a request, decrement the counter. Then run a thread that prints the counter every second. If your counter value is always 1, then you are still running synchronously.

The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.

Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):

You can create a DeepReplace utility that would recursively check and replace keys. Also I'd strongly suggest to only replace value and make sure the key will stay same.

Your selection is just too general, cause it is selecting all

Select the headlines mor specific: