cri-o | Open Container Initiative-based implementation of Kubernetes Container Runtime Interface | Continuous Deployment library

Answer is pretty much here: https://serverfault.com/questions/1064313/ambassador-service-stays-pending . After installing a load balancer the whole setup worked. I decided to go with metallb (https://metallb.universe.tf/installation/#installation-by-manifest for installation). I decided to go with the following configuration for a single-node kubernetes cluster:

PodIP is the local ip of the pod within the cluster. Each pod gets a dynamic IP allocated to it.

You can see the explanation from kubectl command

This looks like a liveness problem related to the CrashLoopBackOff have you cosidered taking a look into this blog it explains very well how to debug the problem blog

As per official documentation from Kubernetes the CRI-O version needs to match your Kubernetes version.

https://v1-19.docs.kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o

So CRI-O 1.19 version should be compatible with the Kubernetes version you want to install.

The way I did a BareMetal setup is by installing METALLB together with ingress-nxginx and used NAT to forward the traffic received on my host (ports 80 & 443) to ingress-nginx.

Thanks to Mariusz K. I found the answer to my problem. In case that someone else might have the same problem here is how I solved it.

First query the cluster (HA) for the etcd members (sample of code):

So the problem is not exactly a bug on CRI-O as we initially thought (also the CRI-O dev team) but it seems to be a lot of configurations that need to be applied if the user desires to use CRI-O as the CRI for kubernetes and also desire to use a private repo.

So I will not put here the configurations for the CRI-O as it is already documented on the ticket that I raised with the team Kubernetes v1.18.2 with crio version 1.18.2 failing to sync with kubelet on RH7 #3915.

The first configuration that someone should apply is to configure the registries of the containers where the images will be pulled:

After a long search I have found a solution. The container kube-flannel-amd64 in the kube-system namespace initially throws the error that it has no permission to access iptables. So the IP packets are not routed over the VXLAN. This caused into the timeout error.

To give the container access to iptables of the hostsystem I changed the official kube-flannel.yml from privileged: false to true.

Holy Hand Grenade of Antioch! I finally fixed it! It only took me, what, about a bazillion years and a restless-night. Sweet Victory! Well... ehm. On to the solution.

I finally understand the comments by @Arghya Sadhu and @Piotr Malec and they were right. I didn't configure my CNI-plugin correctly. I am using Flannel as a network provider and they require a 10.244.0.0/16 subnet. In my crio-bridge.conf found in /etc/cni/net.d/ the default subnet was different (10.85.0.0/16 or something). I thought it would be enough to specify the CIDR on the kubeadm init command but I was wrong. You need to set the correct CIDR in the crio-bridge.conf and podman.conflist (or similar files in the directory). I also thought those files that were installed with CRI-O were configured with reasonable defaults and, to be honest, I didn't fully understand what they were for.

Also something strange happened: According to Flannel the subnet for CRI-O should be /16 but when I checked the logs with journalctl -u kubelet it mentioned a /24 subnet.