prometheus-adapter | An implementation of the custom.metrics.k8s.io API | Analytics library

I'm testing out HPA with custom metrics from application and exposing to K8s using Prometheus-adapter.

My app exposes a "jobs_executing" custom metric that is a numerical valued guage (prometheus-client) in golang exposing number of jobs executed by the app (pod).

Now to cater this in hpa, here is how my HPA configuration looks like:

I am trying to get Kafka topic lag into Prometheus and finally to the APIServer in order to utilize an external metrics HPA for my application.

I am getting the error no metrics returned from external metrics API

Following the guide that google made for deploying an HPA in Google Kubernetes Engine: https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics

And adding the right permissions because I am using Workload Identity with this guide: https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter

And also adding the firewall-rule commented here: https://github.com/kubernetes-sigs/prometheus-adapter/issues/134

I am stuck in a point where the HPA returns me this error:

I have two deployments running a v1 and v2 of the same service in istio. I have set up a custom-metric 'istio-total-requests' to be collected through prometheus adapater.

I have set up an HPA to scale the v2 deployment and can see the target value increasing when I send requests, but what is not happening is the HPA is not scaling the number of pods/replicas.

I am running kubernetes v1.19 on minikube v1.13.1, and cannot understand why its not scaling.

I am using this to export metrics from Prometheus to Kubernetes. That seems to be working. I can (successfully) see the metric names with

For now I have prometheus and prometheus adapter in different namespaces:

I tried to configure adapter YML but I was not successful: apiVersion: apps/v1 kind: Deployment metadata:

To address the default AWS VPC CNI max pod number node limit (Max Pods = Maximum supported Network Interfaces for instance type) * ( IPv4 Addresses per Interface) - i.e. 17 in a t3.medium) in my EKS cluster I started using Weave CNI Plugin to overcome that limitation.

This brings a caveat, if you have an application or container running in the overlay network and the Kubernetes master node / API needs to talk to it, it won't work. For instance, the ApiService v1beta1.metrics.k8s.io tries to connect to metrics-server pods running in the overlay network and it won't succeed. A proposed solution is to run metrics-server with hostNetwork: true which works just fine.

My problems started when we decided to have custom-metrics with prometheus-adapter so that we could have metrics from Kafka consumer group lag for horizontal autoscaling of the consumer pods. To have the ApiService v1beta1.custom.metrics.k8s.io talking with prometheus-adapter pod we also have to set hostNetwork: true but this time the prometheus-adapter cannot access the Prometheus running in the overlay network anymore - and we cannot move everything to the host network!

I'm kind of in a dead-end here. I guess I could use some "tool" to forward the metrics that I need from the overlay network Prometheus to another Prometheus in the host network which would be then used by prometheus-adapter?