kubespray

By default, when Ansible first starts executing a play it will implicitly run the setup module on all the remote hosts involved in the play. This is called the "fact gathering" step, and is controlled by the gather_facts option in the play or the gathering option in ansible.cfg.

The facts gathered in this step include things like operating system flavor, release, information about interfaces and disks, and a variety of other host metadata. The variables are all all in the ansible_* namespace (e.g., ansible_distribution_major_version).

Kubespray is taking advantage of these automatically gathered facts to load the appropriate vars files.

For more information:

• Discovering variables: facts and magic variables (from the Ansible documentation)
• An introduction to Ansible facts

Etc.

The best possible solution I have come up with to date is this, replacing the hard coding with variables is advisable, but it works:

Eventually, I could resolve the issue. The issue was trivial yet bothering. In the Dockerfile, there was a missing script, i.e., npm run build. So, here is the final Dockerfile I used it for building the dist directory along with other requirements:

Running kubectl get ingressclass returned 'No resources found'.

That's the main reason of your issue.

Why?

When you are specifying ingressClassName: nginx in your Grafana values.yaml file you are setting your Ingress resource to use nginx Ingress class which does not exist.

I replicated your issue using minikube, MetalLB and NGINX Ingress installed via modified deploy.yaml file with commented IngressClass resource + set NGINX Ingress controller name to nginx as in your example. The result was exactly the same - ingressClassName: nginx didn't work (no address), but annotation kubernetes.io/ingress.class: nginx worked.

(For the below solution I'm using controller pod name ingress-nginx-controller-86c865f5c4-qwl2b, but in your case it will be different - check it using kubectl get pods -n ingress-nginx command. Also keep in mind it's kind of a workaround - usually ingressClass resource should be installed automatically with a whole installation of NGINX Ingress. I'm presenting this solution to understand why it's not worked for you before, and why it works with NGINX Ingress installed using helm)

In the logs of the Ingress NGINX controller I found (kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx):

Posting comment as the community wiki answer for better visibility

This issue was due to kubelet certificate expired and fixed following these steps. If someone faces this issue, make sure /var/lib/kubelet/pki/kubelet-client-current.pem certificate and key values are base64 encoded when placing on /etc/kubernetes/kubelet.conf

To fix this problem I had to migrate gitlab-secrets.json from /etc/gitlab too, because this file contains the database encryption key, CI/CD variables, and variables used for two-factor authentication.
If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runner lose access to your GitLab server.

Fortunately increasing timeoutSeconds for both livenessProbe & readinessProbe from 1 to 60 fixes the issue.

Based on the comments.

To access setting for different environments (dev, prd) you can do, for example:

Everything seems to be fine, except that your default and variable node_hosts do not match. In my view, if you make them as follows, it will work: