bingbot | A multi account bing bot , from BOTHAT | Bot library

The "paywall notice" does not seem to be recognized in Google's documentation. I am trying to make it visible to all, yet excluded from the page topic and content, without causing cloaking issues. Can I do this in the DOM (for example with the role attribute), or do I need to do it in the JSON-LD markup?

I am implementing a website paywall using client-side JS, with a combination of open graph markup and CSS selectors.

The implementation is based on the programming suggestions by Google at https://developers.google.com/search/docs/advanced/structured-data/paywalled-content

There are 3 types of content on this site, and in this implementation all 3 are rendered by the server for every visitor regardless of paywall status:

1. Free content, visible to all;
2. Paywall notice, not part of the page content/topic, visible only when not logged in; and
3. Paywalled content, visible only to logged in users and search crawlers.

Type 2 is what is causing trouble, and this is not documented by Google.

I am trying to scrape content from a website using Scrapy CrawlSpider Class but I am blocked by the below response. I guess the above error has got to do with the User-Agent of my Crawler. So I had to add a custom Middleware user Agent, but the response still persist. Please I need your help, suggestions on how to resolve this.

I didn't consider using splash because the content and links to be scraped don't have a javascript extension.

My Scrapy spider class:

I am actually working in a company and to improve SEO, i am trying to setup our angular (10) web app with prerender.io to send rendered html to crawlers visiting our website.

The app is dockerized and exposed using an nginx server. To avoid conflict with existing nginx conf (after few try using it), i (re)started configuration from the .conf file provided in the prerender.io documentation (https://gist.github.com/thoop/8165802) but impossible for me to get any response from the prerender service.

I am always facing: "502: Bad Gateway" (client side) and "could not be resolved (110: Operation timed out)" (server side) when i send a request with Googlebot as User-agent.

After building and running my docker image, the website is correctly exposed on port 80. It is fully accessible when i use a web browser, but the error occurs when i try a request as a bot (using curl -A Googlebot http://localhost:80).

To verify if the prerender service correctly receive my request when needed i tried to use an url generated on pipedream.com, but the request never comes.

I tried using different resolver (8.8.8.8 and 1.1.1.1) but nothing changed.

I tried to increase the resolver_timeout to let more time but still the same error.

I tried to install curl in the container because my image is based on an alpine image, curl was successfully installed but nothing changed.

Here is my nginx conf file :

I want to 301 redirect

https://www.example.com/th/test123

to this

https://www.example.com/test123

See above url "th" is removed from url

So I want to redirect all website users to without lang prefix version of url.

Here is my config file

I set up prerender.io for CRA and it works well, but when bot hits URL without parameters it puts in the end of URL - string ".var"

I tried variations of (.*) but it seems not working. Any ideas?

Here is .htaccess file

I would like to block some specific pages from being indexed / accessed by Google. This pages have a GET parameter in common and I would like to redirect bots to the equivalent page without the GET parameter.

Example - page to block for crawlers:

mydomain.com/my-page/?module=aaa

Should be blocked based on the presence of module= and redirected permanently to

mydomain.com/my-page/

I know that canonical can spare me the trouble of doing this but the problem is that those urls are already in the Google Index and I'd like to accelerate their removal. I have already added a noindex tag one month ago and I still see results in google search. It is also affecting my crawl credit.

What I wanted to try out is the following:

I've built an SPA in Angular 2, and I'm hosting it on Firebase Hosting. I have built som extra static html pages specifically for crawl bots (since they do not read updated dynamic html, only the initial index.html) and now I need to rewrite the URL for HTTP requests from bots to these static pages.

I know how to do this in a .htaccess file, but I can't figure out how to translate the rewrite conditions in my firebase.json file.

This is my .htaccess:

I have a large file of user agent strings, and I want to extract one particular section of each request.

For input:

I deploy a Flask app on Heroku. Then, I copy and paste the link to an email in Microsoft Outlook. When I copy and paste the link, exactly 5 get requests are sent to the app. This happens without me clicking the link, before I send the email.

The hostname sending the request is msnbot-157-55-39-74.search.msn.com with the ISP Microsoft Bingbot.

I don't experience the same issue when I copy and paste the link in Gmail.

Why is this happening and how do I prevent this behavior?