crawlers | Some quick 'n dirty web crawlers | Crawler library

I am using Angular Universal for most of my website so that I can pre-render the content for SEO. It is meant to be a public facing site.

I would like to be able to make certain components client-side rendered ONLY to avoid bundling content such as email addresses and social media links from being discoverable by web crawlers.

I used the Angular Universal generated application to create my app. Currently, ALL my components are being rendered server-side. I couldn't find any specific clear example where someone used Angular in an elegant manner to achieve this specific goal. My intent is make my contact info and social media links components completely client-side rendered and added to the DOM at runtime to avoid bots and web crawlers from seeing it.

How do others achieve this without doing something hacky?

I have an application containing videos. Each of the videos has a view count (represented by a property total_views) along with other properties such as its title, uploader, etc. A video view count should be incremented by 1 every time a video is requested/watched. The frontend of this application is a Next.js SPA and the backend is a Lumen/Laravel REST API.

The current REST API backend solution returns the total_views as part of the video entity when the GET /videos/{id} endpoint is called on the API.

I am not sure how to implement video count updates in the most REST generally accepted convention compliant way. I thought of updating the count on a request to GET /videos/{id} but I believe that this is not common REST standards/specification compliant (causing issues with caching, etc.) since the total_views property of the entity object in the response is being updated too. The second option I thought of is using another endpoint such as POST/PUT/PATCH /videos/{id}/views. However, I do not want to use a request body as the backend API should always increment by 1 only (and this way avoiding the client tampering with the view count). Another drawback of this option is that it introduces extra overhead as it requires sending another HTTP request in addition to the GET request for getting the video info.

What are your suggestions?

EDIT: Video view count here might also be seen as page view counts instead of actual video views/plays (Video resource views). Accurate view counting that filters out page views of crawlers, bots, or views where the visitor did not start the video are outside the scope of this question.

The videos are hosted by external 3-rd party hosts and are only embedded via embed codes (iframe) in this application's video web pages.

I have a situation similar to a previous question that uses the following in the accepted answer:

I have successfully run crawlers that read my table in Dynamodb and also in AWS Reshift. The tables are now in the catalog. My problem is when running the Glue job to read the data from Dynamodb to Redshift. It doesnt seem to be able to read from Dynamodb. The error logs contain this

I made a Scrapy code that has 4 crawlers scraping from 4 different E-commerce websites. For each crawler, I want to output 5 products with the lowest prices from each website and export them into a single CSV file.

Right now, my main code looks like this:

I have a ReactJS app built with react-lazy-load-image-component in order to improve performance.

My code wraps components that take time to initialize with something like:

I am trying to create a Lambda@Edge function to return Open Graph HTML for my Angular SPA application. I've installed it into the CloudFrond "Viewer Request" lifecycle. This lambda checks the user agent, and if it's the Facebook or Twitter crawler, it returns HTML (currently hard coded in the lambda for testing). If the request is from any other user-agent, the request is passed through to the origin. The pass-through logic is working properly, but if I try to intercept and return the Open Graph HTML for the crawlers, I get an error.

In CloudWatch, the error reported by CloudFront is:

ERROR Validation error: The Lambda function returned an invalid body, body should be of object type.

In Postman (by faking the user-agent), I get a 502 with:

The Lambda function result failed validation: The body is not a string, is not an object, or exceeds the maximum size.

I'm pulling my hair out with this one. Any ideas? Here's my lambda.

I am using i18n with Angular Universal SSR. The issue is that the client received the text in source locale and after a few seconds are replaced with the correcty locale.

For example, client load http://localhost:4000/en-US/ in the first display shows in es locale and after a few seconds the text is replaced with en-US locale texts.

The build folders are created correctly and the proxy works perfect for each locale. I want the server to return the html with the correct translation so the SEO crawlers can find the content correctly in each locale.

It seems that the problem is in the build, is not generated with the correct locale.

The proyect config in angular.json file:

I am working on a crawler to analyze the internal link structure of websites using a neo4j graph database in combination with spatie crawler.

The idea goes like this:

Whenever a URL is crawled, all Links will be extracted from the DOM. For all links, a node will be created and a relation foundOn->target is added.