prometheus-kubernetes | Monitoring Kubernetes clusters on AWS , GCP and Azure | Dashboard library

I am running the cluster on AWS, the following tutorials work for me https://itnext.io/kubernetes-monitoring-with-prometheus-in-15-minutes-8e54d1de2e13. Let see if you can adapt it. Basicallly, I used helm to install coreos/kube-prometheus.

# initialize tiller account kubectl create serviceaccount -n kube-system tiller kubectl create clusterrolebinding tiller-binding --clusterrole=cluster-admin --serviceaccount kube-system:tiller helm init --service-account tiller # install Prometheus app sleep 1m helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/ helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring helm install coreos/kube-prometheus --name kube-prometheus --namespace monitoring --set global.rbacEnable=true --set prometheus.resources.requests.memory=300Mi # forward ports kubectl port-forward -n monitoring prometheus-kube-prometheus-0 9090 & kubectl port-forward $(kubectl get pods --selector=app=kube-prometheus-grafana -n monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring 3000 & kubectl port-forward -n monitoring alertmanager-kube-prometheus-0 9093 &

Here they use example.io/scrape=true (and similar annotations for specifying the scrape port and the scrape path if it's not /metrics), which is how one achieves the "autodiscovery" part.

If you apply that annotation -- and the relevant config snippets in the Prom config -- to a Service, then Prom will scrape the port and path on the Service, meaning you will have stats for the Service itself, and not the individual Endpoints behind it. Similarly, if you label the Pods, you will gather metrics for the Pods but they would need to be rolled up to have a cross-Pod view of the state of affairs. There are multiple different resource types that can be autodiscovered, including node and ingress, also. They all behave similarly.

Unless you have grave CPU or storage concerns for your Prom instance, I absolutely wouldn't enumerate the scrape targets in the config like that: I would use the scrape annotations, meaning you can change who is scraped, what port, etc. without having to reconfigure Prom each time.

Be aware that if you want to use their example as-is, and you want to apply those annotations from within the kubernetes resource YAML, ensure that you quote the : 'true' value, otherwise YAML will promote that to be a boolean literal, and kubernetes annotations can only be string values.

Applying the annotations from the command line will work just fine:

By digging though the source code I figured out, that Prometheus always uses the in cluster config, if no api_server is provided in the config (discovery/kubernetes/kubernetes.go#L90-L96).

Somehow the docs don't say anything about the Kubernetes configuration parameters, but the source code does (config/config.go#L1026-L1037). Therefore there is not list named api_servers, but a single parameter named api_server.

So your config should look like this (untested):

That sounds like a network issue as the request is timing out. Can you hit that endpoint from inside the Prometheus container?