crawlers | extracting data from various sites | Crawler library

I've created a SPA - Single Page Application with Angular 11 which I'm hosting on a shared hosting server.

The issue I have with it is that I cannot share any of the pages I have (except the first route - /) on social media (Facebook and Twitter) because the meta tags aren't updating (I have a Service which is handling the meta tags for each page) based on the requested page (I know this is because Facebook and Twitter aren't crawling JavaScript).

In order to fix this issue I tried Angular Universal (SSR - Server Side Rendering) and Scully (creates static pages). Both (Angular Universal and Scully) are fixing my issue but I would prefer using the default Angular SPA build.

The approach I am taking:

• Files structure (shared hosting server /public_html/):

We are working with AWS Glue as a pipeline tool for ETL at my company. So far, the pipelines were created manually via the console and I am now moving to Terraform for future pipelines as I believe IaC is the way to go.

I have been trying to work on a module (or modules) that I can reuse as I know that we will be making several more pipelines for various projects. The difficulty I am having is in creating a good level of abstraction with the module. AWS Glue has several components/resources to it, including a Glue connection, databases, crawlers, jobs, job triggers and workflows. The problem is that the number of databases, jobs, crawlers and/or triggers and their interractions (i.e. some triggers might be conditional while others might simply be scheduled) can vary depending on the project, and I am having a hard time abstracting this complexity via modules.

I am having to create a lot of for_each "loops" and dynamic blocks within resources to try to render the module as generic as possible (e.g. so that I can create N number of jobs and/or triggers from the root module and define their interractions).

I understand that modules should actually be quite opinionated and specific, and be good at one task so to speak, which means my problem might simply be conceptual. The fact that these pipelines vary significantly from project to project make them a poor use case for modules.

On a side note, I have not been able to find any robust examples of modules online for AWS Glue so this might be another indicator that it is indeed not the best use case.

Any thoughts here would be greatly appreciated.

EDIT: As requested, here is some of my code from my root module:

Good day.

We have a web portal coded in vanilla PHP that has a blog section where [mysite.com/blog.php?blog=1] outputs the content of the desired file.

This led to our SEO expert pointing out that it is a poorly formatted URL for SEO.

We then decided to use .htaccess to display named URLs

blog=Residential_Relocation -> blogs.php?blog=1 to output [mysite.com/blog.php?blog=Residential_Relocation]

But now it is seen as a duplicate.

How can we go about to only read the file from the blog=1 URL without it being picked up by crawlers?

Frameworks like React or Vue make use of DOM manipulation for rendering components dynamically withing index.html. Additionally, SPA routers generate virtual routes (usually prefixed with "#") simulating the legacy SSR pages architecture.

Now the question is, can crawlers read within the SPA javascript links to these virtual routes?

Here's my code:

In my first attempt at using Cloud to deploy an app...

The problem: GCP (Google Cloud Platform) unexpected instance hour usage (Frontend Instance Hours). High traffic was not the issue but for some reason a bunch of "instances" and "versions" were created by their autoscaling feature.

Solution they suggested: Disable autoscaling and stop serving previously deployed versions of your instance. I still need one version/instance running but through their console I still have not found where it shows how many versions/instances I have running or where to stop them (also verifying that at least 1 instance is still working in order to not break my app)

My app is simple app that was developed by Google developers and recommended by them for dynamic rendering a JS SPA (allows search engines and crawlers to see fully rendered html).
My actual website together with a node app to point to GCP for crawlers is hosted else where (on Godaddy) and both are working together nicely.

The app I deployed to GCP is called Rendertron (https://github.com/GoogleChrome/rendertron)

Google also recommends deploying to GCP (most documentation covers that form of deployment). I attempted deploying to my Godaddy shared hosting and it was not straight forward and easy to make work so I simply attempted creating a GCP project and tried deploying there. All worked great!

After deploying the app to GCP that has almost no traffic yet, I expected zero costs or at most something under a dollar.

Unfortunately, I received a bill for more than $150 for the month with approx the same projected for the next month.

Without paying an addition $150 for tech support, I was able to contact GCP billing over the phone and they are great in that they are willing to reimburse the charges but only after I resolve the problem myself.

They are generous with throwing a group of document links at you (common causes of unexpected instance hour usage) but can't help further than that.

After many google searches, reading through documentation, paying for and watching gcloud tutorials through pluralsight.com, the direction I have understood or not understood so far is as follows:

• almost all documentation, videos and tutorials talk about managing or turning off autoscaling using Compute Engine Instance Groups
• It is not clear that instance groups is not another hole I will fall into that is a paid service and I will be charged more than necessary
• Instance groups seems like overkill for a simple app that wants only one instance running at minimal cost
• there is not enough or difficult to find documentation for how to run a very small scale app at minimal cost using minimal resources
• I have not read or watched anything yet of how to simply use the config .yaml file (initially deployed) to make sure the app does not autoscale and also if I find that it seems like I still need to delete versions or instances that have already been started and it is not clear in how to do that as well.
• Instances and Versions are not clear on google console of how many are running, I still have not found on google console where there are multiple instances/versions running.

I can use a direction to continue my attempt of investigating how to resolve the issue.

1. The direction of me needing to create a Group Instance (so I can manage the no autoscaling from there) is the way to go and where I should focus my attempts?

2. The direction of continuing learning how to simply update my config in the .yaml file to create no scaling, for example something like setting both min_instances and max_instances to 1 together with learning how to manually stop (directly from GCP console) more than 1 instance/version that are currently running is where I should focus on?

3. A third option?

As a side note, autoscaling with GCP does not seem very intelligent.
Why would my app that has almost no traffic run into an issue that multiple instances were created?

Any insight will be greatly appreciated.

**** Update **** platform info

My app is deployed to Google App Engine (GAE) (deployed code, not a container)

Steps taken for Deploy:

We are using the Apify Web Scraper actor to create a URL validation task that returns the input URL, the page's title, and the HTTP response status code. We have a set of 5 test URLs we are using: 4 valid, and 1 non-existent. The successful results are always included in the dataset, but never the failed URL.

Logging indicates that the pageFunction is not even reached for the failed URL:

I have a big CSV text file uploaded weekly to an S3 path partitioned by upload date (maybe not important). The schema of these files are all the same, the formatting is all the same, the naming conventions are all the same. Each file contains ~100 columns and ~1M rows of mixed text/numeric types. The raw data looks like this:

When we make a stripe checkout session we include a success url: