user-agents | JavaScript library for generating random user agents

I'm making a scraper to read question / answer data for students that supports RDFa, Json LD, and Microdata, but Quora confuses me. I need to understand how it's read so that I can read it in my HTML question / answer scraper for situations like this.

In a google search, I see a QA block, but if I go to the URL https://www.quora.com/What-happens-when-sodium-chloride-and-water-is-heated-to-dry I don't see any evidence of JSON LD, RDFa or Microdata. How is google reading quora's question / answer information?

Possible reasons I can think of:

• They only show that data to search engine user-agents. So perhaps I should change the user-agent to a scraper when requesting the page.
• Google figured it out on its own. This means I need to create some NLP solution to get the information.
• Key words that identify the page as question / answer.
• Google does something special for big Q/A sites like quora (but stack overflow has schema.org, so I don't think this is true).

PS: Even google doesn't show support for other formats: https://developers.google.com/search/docs/advanced/structured-data/qapage

Title says it really - i can build locally but it fails to deploy. Has an issue with Timeout - not sure how to force this to work?

Here's my package.json, and the logs afterwards.

I tried adding Timeout by running a yarn install but then it triggered some other dependency issues with node-gyp

Many thanks for any tips!

Package.json

I am trying to run multiple requests at the same time with a proxy. I want a proxy to initiate one request at a time, that's why I have a gethandlerindexed function

Also, the error occurs during the last call to gethandlerindexed, here max = 2, so I have 2 proxies, 2 user agents but it doesn't work. The error in question is: Empty queue

I do not understand where this error can come from.

Do the big guys (Google, Microsoft, etc...) remember all HTTP clients and more importantly, the User-Agents that connected to them?

If so, should you implement this as a startup? (make your server remember the clients)

I'm not asking for advice, only for practicality or if there's some protocol somewhere that requires it. Like what's the standard, not your opinion.

I have a web scraping script that has recently ran into a 403 error. It worked for a while with just the basic code but now has been running into 403 errors. I've tried using user agents to circumvent this and it very briefly worked, but those are now getting a 403 error too.

Does anyone have any idea how to get this script running again?

If it helps, here is some context: The purpose of the script is to find out which artists are on which Tidal playlists, for the purpose of this question - I have only included the snippet of code that gets the site as that is where the error occurs.

Thanks in advance!

The basic code looks like this:

UPDATE: it seems that the user-agent isn't the only header some hosts require to serve HTML, I also had to add the 'accepts' header, in the end this solved the problem for me with many hosts:

I have a scrapy crawler that scrapes data off a website and uploads the scraped data to a remote MongoDB server. I wanted to host it on heroku to scrape automatically for a long time. I am using scrapy-user-agents to rotate between different user agents. When I use scrapy crawl locally on my pc, the spider runs correctly and returns data to the MongoDB database.

However, when I deploy the project on heroku, I get the following lines in my heroku logs :

2020-12-22T12:50:21.132731+00:00 app[web.1]: 2020-12-22 12:50:21 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying https://indiankanoon.org/browse/> (failed 1 times): 503 Service Unavailable

2020-12-22T12:50:21.134186+00:00 app[web.1]: 2020-12-22 12:50:21 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36

(it fails similarly for 9 times until:)

2020-12-22T12:50:23.594655+00:00 app[web.1]: 2020-12-22 12:50:23 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying https://indiankanoon.org/browse/> (failed 9 times): 503 Service Unavailable

2020-12-22T12:50:23.599310+00:00 app[web.1]: 2020-12-22 12:50:23 [scrapy.core.engine] DEBUG: Crawled (503) https://indiankanoon.org/browse/> (referer: None)

2020-12-22T12:50:23.701386+00:00 app[web.1]: 2020-12-22 12:50:23 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <503 https://indiankanoon.org/browse/>: HTTP status code is not handled or not allowed

2020-12-22T12:50:23.714834+00:00 app[web.1]: 2020-12-22 12:50:23 [scrapy.core.engine] INFO: Closing spider (finished)

In summary, my local IP address is able to scrape the data while when Heroku tries, it is unable to. Can changing something in the settings.py file correct it?

My settings.py file :

I would like to check in my Google Analytics how big a proportion of my user base is using Huawei devices which are no longer using google services but instead using App Gallery etc.

I was wondering if I could e.g. look for specific OS versions in the User-Agents etc.?

I'm facing a very strange error from few days now. I have a python2.7 project that was running smoothly but since few days its been throwing an error:

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

I'm using virtual environment for my project. What happened was that few days ago I tried installing nginx using brew command and what I believe is brew updated some dependencies that were being used for python2.7 project (this is what i think might be the case). Now since that day, I'm facing this issue and I have googled it everywhere but couldn't resolve. Below is some information you might need to figure out.

my requirements.txt file