autoscaler | Automatically scale the capacity of your Spanner instances | Monitoring library

I have a web app (react + node.js) running on App Engine.
I would like to kick off (from this web app) a Machine Learning job that requires a GPU (running in a container on AI platform or running on GKE using a GPU node pool like in this tutorial, but we are open to other solutions).
I was thinking of trying what is described at the end of this answer, basically making an HTTP request to start the job using project.job.create API.

More details on the ML job in case this is useful: it generates an output every second that is stored on Cloud Storage and then read in the web app.

I am looking for examples of how to set this up? Where would the job configuration live and how should I set up the API call to kick off that job? Are the there other ways to achieve the same result?

Thank you in advance!

We have setup a GKE cluster using Terraform with private and shared networking:

Network configuration:

I'm running a google cloud composer GKE cluster. I have a default node pool of 3 normal CPU nodes and one nodepool with a GPU node. The GPU nodepool has autoscaling activated.

I want to run a script inside a docker container on that GPU node.

For the GPU operating system I decided to go with cos_containerd instead of ubuntu.

I've followed https://cloud.google.com/kubernetes-engine/docs/how-to/gpus and ran this line:

I'm trying to set up FluentBit for my EKS cluster in Terraform, via this module, and I have couple of questions:

cluster_identity_oidc_issuer - what is this? Frankly, I was just told to set this up, so I have very little knowledge about FluentBit, but I assume this "issuer" provides an identity with needed permissions. For example, Okta? We use Okta, so what would I use as a value in here?

cluster_identity_oidc_issuer_arn - no idea what this value is supposed to be.

worker_iam_role_name - as in the role with autoscaling capabilities (oidc)?

This is what eks.tf looks like:

I have created a k8s cluster with kops (1.21.4) on AWS and as per the docs on autoscaler. I have done the required changes to my cluster but when the cluster starts, the cluster-autoscaler pod is unable to schedule on any node. When I describe the pod, I see the following:

According to the K8s documentation, to avoid flapping of replicas property stabilizationWindowSeconds can be used

The stabilization window is used to restrict the flapping of replicas when the metrics used for scaling keep fluctuating. The stabilization window is used by the autoscaling algorithm to consider the computed desired state from the past to prevent scaling.

When the metrics indicate that the target should be scaled down the algorithm looks into previously computed desired states and uses the highest value from the specified interval.

From what I understand from documentation, with the following hpa configuration:

I can use GKE Autopilot to run arbitrary workloads on a sandbox project (with default networks, default service account, default firewall rules) just fine.

But I need to create a GKE Autopilot cluster in an existing project which isn't using the default settings for a few different things like networking and when I try, the pods never get run. My problem lies in identifying the underlying reason for the failure and which part of project setup is preventing GKE Autopilot to work.

The error messages and logs are very very scarse. The only things that I see are:

• in the workloads UI, for my pod, it says "Pod unschedulable"
• in the pod UI, under events, it says "no nodes available to schedule pods" and "pod triggered scale-up: [{...url-of-an-instance-group...}]"
• under the cluster autoscaler logs, there is a "scale.up.error.waiting.for.instances.timeout" buried in a resultInfo log (with a reference to a instance group url)

I can't find anything online about why the scaling up would fail in the Autopilot mode which is supposed to be such a hands-off experience. I understand I'm not giving much details about the pod specification (any would fail!) or my project settings, but simply where to look next would be helpful in my current situation.

I am running a GPU intensive workload on demand on GKE Standard, where I have created the appropriate node pool with minimum 0 and maximum 5 nodes. However, when a Job is scheduled on the node pool, GKE presents the following error:

I have a GKE cluster which doesn't scale up when a particular deployment needs more resources. I've checked the cluster autoscaler logs and it has entries with this error: no.scale.up.nap.pod.zonal.resources.exceeded. The documentation for this error says:

Node auto-provisioning did not provision any node group for the Pod in this zone because doing so would violate resource limits.

I don't quite understand which resource limits are mentiond in the documentation and why it prevents node-pool from scaling up?

If I scale cluster up manually - deployment pods are scaled up and everything works as expected, so, seems it's not a problem with project quotas.