calico | Cloud native networking and network security | Networking library

I have recreated the situation according to the attached tutorial and it works for me. Make sure, that you are trying properly login:

To protect your cluster data, Dashboard deploys with a minimal RBAC configuration by default. Currently, Dashboard only supports logging in with a Bearer Token. To create a token for this demo, you can follow our guide on creating a sample user.

Warning: The sample user created in the tutorial will have administrative privileges and is for educational purposes only.

You can also create admin role:

The example from the article you mentioned has it: spec.interfaceName: eth0. Have you tried it so far?

For each host point that you want to secure with policy, you must create a HostEndpoint object. To do that, you need the name of the Calico node on the host that owns the interface; in most cases, it is the same as the hostname of the host.

In the following example, we create a HostEndpoint for the host named my-host with the interface named eth0, with IP 10.0.0.1. Note that the value for node: must match the hostname used on the Calico node object.

When the HostEndpoint is created, traffic to or from the interface is dropped unless policy is in place.

the port is overwriten by the dns service to 8053. the tcpdumper is running inside the pod, so it does not know it is re-routed .

I found the answer to my question here: https://www.stackrox.io/blog/kubernetes-networking-demystified/ There probably is a caveat that this may vary to some extent depending on which networking CNI you are using, although everything I saw was strictly related to Kubernetes itself.

I'm still trying to digest the content of this blog, and I highly recommend referring directly to that blog, instead of relying on my answer, which could be a poor retelling of the story.

Here is approximately how a package that arrives on port 443 flows.

You will need to use the command to see the tables.

Calico is only used for Network Policies in GKE. By default GKE uses a Google Network Plugin. You also have the option to enable Dataplane V2 which is eBPF Based.

In both cases the Plugins are managed by Google and you cannot change them

The cause of the problem was an update of the cluster to the kubernetes version 1.21 while my cluster was meeting the following conditions:

• private and public service endpoint enabled
• VRF disabled

In Kubernetes version 1.21, Konnectivity replaces OpenVPN as the network proxy that is used to secure the communication of the Kubernetes API server master to worker nodes in the cluster.
When using Konnectivity, a problem exists with masters to cluster nodes communication when all of the above mentioned conditions are met.

• disabled the private service endpoint (the public one seems not to be a problem) by using the command
  ibmcloud ks cluster master private-service-endpoint disable --cluster (this command is provider specific, if you are experiencing the same problem with a different provider or on a local installation, find out how to disable that private service endpoint)
• refreshed the cluster master using ibmcloud ks cluster master refresh --cluster and finally
• reloaded all the worker nodes (in the web console, should be possible through a command as well)
• waited for about 30 minutes:
  • Dashboard available / reachable again
  • Pods accessible and schedulable again

BEFORE you update any cluster to kubernetes 1.21, check if you have enabled the private service endpoint. If you have, either disable it or delay the update until you can, or enable VRF (virtual routing and forwarding), which I couldn't but was told it was likely to resolve the issue.

Running kubectl get ingressclass returned 'No resources found'.

That's the main reason of your issue.

Why?

When you are specifying ingressClassName: nginx in your Grafana values.yaml file you are setting your Ingress resource to use nginx Ingress class which does not exist.

I replicated your issue using minikube, MetalLB and NGINX Ingress installed via modified deploy.yaml file with commented IngressClass resource + set NGINX Ingress controller name to nginx as in your example. The result was exactly the same - ingressClassName: nginx didn't work (no address), but annotation kubernetes.io/ingress.class: nginx worked.

(For the below solution I'm using controller pod name ingress-nginx-controller-86c865f5c4-qwl2b, but in your case it will be different - check it using kubectl get pods -n ingress-nginx command. Also keep in mind it's kind of a workaround - usually ingressClass resource should be installed automatically with a whole installation of NGINX Ingress. I'm presenting this solution to understand why it's not worked for you before, and why it works with NGINX Ingress installed using helm)

In the logs of the Ingress NGINX controller I found (kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx):

1. I think you need to restart K3s via systemd if you want your cluster back after kill. Try command:
   sudo systemctl restart k3s This is supported by the installation script for systemd and openrc. Refer rancher doc

2. The pod-xxx id will remain same as k3s-killall.sh doesn't uninstall k3s (you can verify this, after k3s-killall script k3s -v will return output) and it only restart the pods with same image. The Restarts column will increase the count of all pods.

For those of you running into a similar issue with your IPv6 only Kubernetes clusters heres what I have investigated found so far.

Background: It seems that this is a generic issue relating to IPv6 and CRI. I was running containerd in my setup and containerd versions 1.5.0-1.5.2 added two PRs (don't use socat for port forwarding and use happy-eyeballs for port-forwarding) which fixed a number of issues in IPv6 port-forwarding.

Potential fix: Further to pulling in containerd version 1.5.2 (as part of Ubuntu 20.04 LTS) I was also getting the error IPv4: dial tcp4 127.0.0.1:15021: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found when port-forwarding. This is caused by a DNS issue when resolving localhost. Hence I added localhost to resolve as ::1 in the host machine with the following command.