kube-state-metrics | Add-on agent to generate and expose cluster-level metrics | Analytics library

This is because you're targeting the wrong service. You're using the alert manager url instead of the prometheus server.
The URL should be this one :

an operator pod acts as a controller that listens to events regarding specific custom resources. if you only deploy the operator, you have to seperately deploy the custom resource you wish to be created.

with the prometeus-operator, that would be a custom resource of kind "prometheus". if the helm chart you choose is capable to also deploy this (or not) should be indicated in the charts values.yaml and documented on their github page.

you can also use the examples from the prometheus-operator repo to create prometheus instances. check out these files to do so: https://github.com/prometheus-operator/prometheus-operator/tree/main/example/rbac/prometheus

The problem is that you are doing those operations at the wrong time. relabel_configs happens before metrics are actually gathered, so, at this time, you can only manipulate the labels that you got from service discovery.

That node label comes from the exporter. Therefore, you need to do this relabeling action under metric_relabel_configs:

Enable persistence if you want to make it stateful. However I did not see an option to make Grafana a statefulset in the chart you mentioned.

Usually you will see persistence enable option if the corresponding Helm chart support it. For example: you can enable persistence in this grafana helm chart. You may generate template out of it and make use of it in your repo.

The solution for the issue is to regenerate the kubeconfig file for the admin:

You can check for another instance of Prometheus running on the same cluster:

I found this guide very helpful.

Please keep in mind that depending on the prometheus stack you are using labels and names can have different default values (for me, using kube-prometheus-stack, for example the secret name was prometheus-kube-prometheus-stack-prometheus instead of prometheus-k8s).

Essential quotes:

ServiceMonitor objects and the namespace where they belong are selected by the serviceMonitorSelector and serviceMonitorNamespaceSelectorof a Prometheus object. The name of a ServiceMonitor is encoded in the Prometheus configuration, so you can simply grep whether it is present there. The configuration generated by the Prometheus Operator is stored in a Kubernetes Secret, named after the Prometheus object name prefixed with prometheus- and is located in the same namespace as the Prometheus object. For example for a Prometheus object called k8s one can find out if the ServiceMonitor named my-service-monitor has been picked up with:

You can use on() operator:

According to the link that you shared, {namespace} and {pod} are default labels provided in the metrics, they are referring to the exposed metrics included in the kube-state-metrics (KSM) service.

kube-state-metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. The exposed metrics are detailed in this document.

In the following links, you can find the related metric for Pods and namespace.

Speaking about the default labels for pods, you need to create a Pod label controller or indicate the label in the Pod Template.

If you don't explicitly specify labels for the controller, Kubernetes will use the pod template label as the default label for the controller itself. The pod selector will also default to pod template labels if unspecified.

If you want to know more about best practices for labels, please follow this link. If you want to know more about Labels and selector, follow this link. More about Pod Template here.