prometheus-operator | Prometheus Operator creates/configures/manages Prometheus | Monitoring library

an operator pod acts as a controller that listens to events regarding specific custom resources. if you only deploy the operator, you have to seperately deploy the custom resource you wish to be created.

with the prometeus-operator, that would be a custom resource of kind "prometheus". if the helm chart you choose is capable to also deploy this (or not) should be indicated in the charts values.yaml and documented on their github page.

you can also use the examples from the prometheus-operator repo to create prometheus instances. check out these files to do so: https://github.com/prometheus-operator/prometheus-operator/tree/main/example/rbac/prometheus

As commented by @jan-garaj there is no need to query all the endpoints. All return the accumulated data of all realms. So it is enough to just scrape the endpoint of one realm (e.g. the master realm).

Thanks a lot!

Enable persistence if you want to make it stateful. However I did not see an option to make Grafana a statefulset in the chart you mentioned.

Usually you will see persistence enable option if the corresponding Helm chart support it. For example: you can enable persistence in this grafana helm chart. You may generate template out of it and make use of it in your repo.

You can check for another instance of Prometheus running on the same cluster:

Assuming that the Prometheus pods did start, their collective hostnames would be found via service discovery like prometheus.monitoring.svc.cluster.local

https://cloud.google.com/kubernetes-engine/docs/concepts/service-discovery

You might also be interesting in exposing Prometheus itself

I found this guide very helpful.

Please keep in mind that depending on the prometheus stack you are using labels and names can have different default values (for me, using kube-prometheus-stack, for example the secret name was prometheus-kube-prometheus-stack-prometheus instead of prometheus-k8s).

Essential quotes:

ServiceMonitor objects and the namespace where they belong are selected by the serviceMonitorSelector and serviceMonitorNamespaceSelectorof a Prometheus object. The name of a ServiceMonitor is encoded in the Prometheus configuration, so you can simply grep whether it is present there. The configuration generated by the Prometheus Operator is stored in a Kubernetes Secret, named after the Prometheus object name prefixed with prometheus- and is located in the same namespace as the Prometheus object. For example for a Prometheus object called k8s one can find out if the ServiceMonitor named my-service-monitor has been picked up with:

Karans-MacBook-Pro:prometheus-yamls karanalang$ telnet 10.76.0.11 30900 Trying 10.76.0.11...

Karans-MacBook-Pro:prometheus-yamls karanalang$ ping 10.76.0.7 PING 10.76.0.7 (10.76.0.7): 56 data bytes

The IP that you used above appeared to be in the Pod CIDR range when judged from the EndPoints result in the question. These are not the worker node IP, which means you need to first check if you can reach any of the worker node over the network that you reside now (home? vpn? internet?), and the worker node already has the correct port (30900) opened.

Try using 'matchers' instead of 'match' which supports negative matching: Instead of