scraps | Miscellaneous stuff that does n't fit

It wouldnt help like you describe as Instagram always shows the first comments first, so you would just get 10x the same comments. If you need more and want to stick with Apify try this actor: https://apify.com/jaroslavhejlek/instagram-scraper - it is more complex but once you set it, it will scrape many more comments than the one you mention. Important things: For scraping more comments you need use cookies (how to is described in the readme) - it is much better to use a dummy account for it and not your real one which you want to keep. Also you need to set proxies for automatic and NOT use residentials for it. Including pic of what you should add to the input: input of the actor

The correct approach for code like this would be something like:

so I figured that you are trying to monitor your application by letting prometheus scrape a metrics exporter for it. this metrics exporter runs in another pod on the same cluster as your application pod does, but because the metrics exporter runs on another pod, it has to find the applications pod to be able to generate the metrics for it.

I propose that you follow the kubernetes sidecar pattern and let the metrics exporter run in the same pod as your applications container. this way you avoid the need to let the metrics exporter discover your applications pod because it already knows where it runs, because containers in the same pod discover each other via localhost and their individual ports. this way, you can always predict where your application runs and you can configure your metrics exporter reliably in the same way for each application pod.

check https://kubernetes.io/docs/concepts/workloads/pods/#workload-resources-for-managing-pods to see how containers in the same pod interact with each other and how the sidecar pattern works.

It is nearly the same, define your empty list to store results of specialUrls and iterate over your initial result list of urls:

Try to add \n in f.write() so your string h will write to new line

From this error it should not be hard to guess that headline doesn't contain any text. This is due to get_headline never returning anything - it just print s the text. so headline is None.

Use the code Below

From what I can see, div.autoload_continue does not contain any link. It is like a button that, if you click it, it requests with JavaScript. You can check the endpoint of the request by viewing in Devtools > Networks.

Here is what I see: the website requests this for first load https://economictimes.indiatimes.com/lazyloadlistnew.cms?msid=3053611&curpg=1&img=1.
Then if I scroll down it requests
https://economictimes.indiatimes.com/lazyloadlistnew.cms?msid=3053611&curpg=2&img=1
When I click load more it requests
https://economictimes.indiatimes.com/lazyloadlistnew.cms?msid=3053611&curpg=3&img=0

Look at the parameter curpg, which is increasing; it indicates the page. You may just iterate numbers to change the curpg params.
img param is a toggle to display the image.
As for msid param, it's the id for article list. You can find that value from meta in head

You have 3 problems:

1. Change response.request.meta to response.meta.get
2. yield scrapy.Request(url=response.urljoin(books),callback=self.book_info) , look at the 'books' url and see why you can't join them, you should change it to response.follow(url=books, callback=self.book_info
3. You forgot to pass the meta data to 'book_info' function.