netshoot | Kubernetes network trouble-shooting swiss | Networking library

So I seem to have fixed the issue. Turns out to be an issue with the OS (Arch Linux) not docker, I found this out by running the same files on a Windows system.

For any arch users I've listed my steps out at https://bbs.archlinux.org/viewtopic.php?pid=1973008#p1973008

Problem with package.json copying solved

server/Dockerfile

First of all, according to the docs - please note that Calico and kubeadm support Centos/RHEL 7+.
In both Calico and kubeadm documentation we can see that they only support RHEL7+.

By default RHEL8 uses nftables instead of iptables ( we can still use iptables but "iptables" on RHEL8 is actually using the kernel's nft framework in the background - look at "Running Iptables on RHEL 8").

9.2.1. nftables replaces iptables as the default network packet filtering framework

I believe that nftables may cause this network issues because as we can find on nftables adoption page:

Kubernetes does not support nftables yet.

Note: For now I highly recommend you to use RHEL7 instead of RHEL8.

With that in mind, I'll present some information that may help you with RHEL8.
I have reproduced your issue and found a solution that works for me.

• First I opened ports required by Calico - these ports can be found here under "Network requirements".
  As workaround:
• Next I reverted to the old iptables backend on all cluster nodes, you can easily do so by setting FirewallBackend in /etc/firewalld/firewalld.conf to iptables as described
  here.
• Finally I restarted firewalld to make the new rules active.

I've tried nslookup from Pod running on worker node (kworker) and it seems to work correctly.

Whats is your result of ping 8.8.8.8 in container? or whats is your commend when you want to run container.

You've published the port, which forwards the port from the docker host to the container. Therefore you want to connect to http://localhost:3000. Connecting to container IP may fail with desktop installs since docker runs inside of a VM and those private IP's are only visible in the VM.

If you happen to be running docker-machine (this is the case with older docker toolbox installs), then you'll need to get the IP of the VM. Run echo $DOCKER_HOST to see the IP address and adjust the port to port 3000.

tcpdump has an option to send raw captured packets to stdout, send it to a file on host:

I believe you are misinterpreting the error.

Could not find address for mongo-2.mongo:27017: SocketException: Host not found (authoritative)"

The pod is created with an IP attached. Then it's registered into DNS:

Pod-0 has the IP 10.0.0.10 and now it's FQDN is Pod-0.servicename.namespace.svc.cluster.local

Pod-1 has the IP 10.0.0.11 and now it's FQDN is Pod-1.servicename.namespace.svc.cluster.local

Pod-2 has the IP 10.0.0.12 and now it's FQDN is Pod-2.servicename.namespace.svc.cluster.local

But DNS is a live service, IPs are dynamically assigned and can't be duplicated. So whenever it receives a request:

"Connect me with Pod-A.servicename.namespace.svc.cluster.local"

It tries to reach the registered IP and if the Pod is offline due to a rolling update, it will think the pod is unavailable and will return "Could not find the address (IP) for Pod-0.servicename" until the pod is online again or until the IP reservation expires and only then the DNS registry will be recycled.

The DNS is not discarting the DNS name registered, it's only answering it's currently offline.

You can either ignore the errors during the rolling or rethink your script and try using the internal js environment as mentioned in the comments for continuous monitoring of the mongo status.

EDIT:

• When Pods from a StatefulSet with N replicas are being deployed, they are created sequentially, in order from {0..N-1}.
• When Pods are being deleted, they are terminated in reverse order, from {N-1..0}.
• This is the expected/desired default behavior.
• So the error is expected, since the rollingUpdate makes the pod temporarily unavailable.