cloud-provider-vsphere | Kubernetes Cloud Provider for vSphere https | Azure library

I have a network issue on my cluster and at first I thought it was a routing issue but discovered that maybe the outgoing packet from the cluster isn't getting wrapped with the node ip when leaving the node.

Background is that I have two clusters. I set up the first one (months ago) manually using this guide and it worked great. Then the second one I built multiple times as I created/debugged anisble scripts to automate how I created the first cluster.

On cluster2 I have the network issue... I can get to pods on other nodes but can't get to anything on my regular network. I have tcpdump'd the physical interface on node0 in cluster2 when pinging from a busybox pod and the 172.16.0.x internal pod ip is visible at that interface as the source ip - and my network outside the node has no idea what to do with it. But on cluster1 this same test shows the node ip in place of the pod ip - which is how I assume it should work.

My question is how can I troubleshoot this? Any ideas would be great as I have been at this for several days now. Even if it seems obvious as I can no longer see the forest through the trees... ie. both clusters look the same everywhere I know how to check :)

caveat to "my clusters are the same": Cluster1 is running kubectl 1.16 cluster2 is running 1.18

----edit after @Matt dropped some kube-proxy knowledge on me----

Did not know that kube-proxy rules could just be read by iptables command! Awesome!

I think my problem is those 10.net addresses in the broke cluster. I don't even know where those came from as they are not in any of my ansible config scripts or kube init files... I use all 172's in my configs.

I do pull some configs direct from source (flannel and CSI/CPI stuff) I'll pull those down and inspect them to see if the 10's are in there... Hopefully it's in the flannel defaults or something and I can just change that yml file!

cluster1 working: