kubes | kubes | Command Line Interface library

The problem is that you are doing those operations at the wrong time. relabel_configs happens before metrics are actually gathered, so, at this time, you can only manipulate the labels that you got from service discovery.

That node label comes from the exporter. Therefore, you need to do this relabeling action under metric_relabel_configs:

Amazon VPC CNI and Flannel cannot co-exist on EKS. Note Flannel is not on the suggested alternate compatible CNI. To get an idea what does it take to use Flannel on EKS checkout this excellent blog.

You can use the k8sutil repo, see the apply example:

Running kubectl get ingressclass returned 'No resources found'.

That's the main reason of your issue.

Why?

When you are specifying ingressClassName: nginx in your Grafana values.yaml file you are setting your Ingress resource to use nginx Ingress class which does not exist.

I replicated your issue using minikube, MetalLB and NGINX Ingress installed via modified deploy.yaml file with commented IngressClass resource + set NGINX Ingress controller name to nginx as in your example. The result was exactly the same - ingressClassName: nginx didn't work (no address), but annotation kubernetes.io/ingress.class: nginx worked.

(For the below solution I'm using controller pod name ingress-nginx-controller-86c865f5c4-qwl2b, but in your case it will be different - check it using kubectl get pods -n ingress-nginx command. Also keep in mind it's kind of a workaround - usually ingressClass resource should be installed automatically with a whole installation of NGINX Ingress. I'm presenting this solution to understand why it's not worked for you before, and why it works with NGINX Ingress installed using helm)

In the logs of the Ingress NGINX controller I found (kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx):

Posting comment as the community wiki answer for better visibility

This issue was due to kubelet certificate expired and fixed following these steps. If someone faces this issue, make sure /var/lib/kubelet/pki/kubelet-client-current.pem certificate and key values are base64 encoded when placing on /etc/kubernetes/kubelet.conf

Unfortunately, I don't have access to Azure AKS, so I've reproduced this issue on my GKE cluster. Below I'll provide some explanations that may help to resolve your problem.

First you can try to execute this node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate rule to see if it returns any result:

If it doesn't return any records, please read the following paragraphs.

Rather than creating a completely new scrape configuration for cadvisor, I would suggest using one that is generated by default when kubelet.serviceMonitor.cAdvisor: true, but with a few modifications such as changing the label to job=kubernetes-cadvisor.

In my example, the 'kubernetes-cadvisor' scrape configuration looks like this:

NOTE: I added this config under the additionalScrapeConfigs in the values.yaml file (the rest of the values.yaml file may be like yours).

Fortunately increasing timeoutSeconds for both livenessProbe & readinessProbe from 1 to 60 fixes the issue.

Everything seems to be fine, except that your default and variable node_hosts do not match. In my view, if you make them as follows, it will work:

The issue with default kernel settings was a bug in kOps. The installed did not set the sysctl settings that kubelet expects.

The issue with the admission controller is simply a missing admission controller configuration file.