ingress-nginx | Ingress-NGINX Controller for Kubernetes | Load Balancing library

I have installed Grafana, Loki, Promtail and Prometheus with the grafana/loki-stack.

I also have Nginx set up with the Nginx helm chart.

Promtail is ingesting logs fine into Loki, but I want to customise the way my logs look. Specifically I want to remove a part of the log because it creates errors when trying to parse it with either logfmt or json (Error: LogfmtParserErr and Error: JsonParserErr respectively).

The logs look like this:

I have a simple ingress configuration file-

I need help understanding in detail how an ingress controller, specifically the ingress-nginx ingress controller, is supposed to work. To me, it appears as a black box that is supposed to listen on a public IP, terminate TLS, and forward traffic to a pod. But exactly how that happens is a mystery to me.

The primary goal here is understanding, the secondary goal is troubleshooting an immediate issue I'm facing.

I have a cluster with five nodes, and am trying to get the Jupyterhub application to run on it. For the most part, it is working fine. I'm using a pretty standard Rancher RKE setup with flannel/calico for the networking. The nodes run RedHat 7.9 with iptables and firewalld, and docker 19.03.

The Jupyterhub proxy is set up with a ClusterIP service (I also tried a NodePort service, that also works). I also set up an ingress. The ingress sometimes works, but oftentimes does not respond (connection times out). Specifically, if I delete the ingress, and then redeploy my helm chart, the ingress will start working. Also, if I restart one of my nodes, the ingress will start working again. I have not identified the circumstances when the ingress stops working.

Here are my relevant services:

I am following this official k8 ingress tutorial. However I am not able to curl the minikube IP address and access the "web" application.

I have installed using instructions at this link for the Install NGINX using NodePort option.

When I do ks logs -f ingress-nginx-controller-7f48b8-s7pg4 -n ingress-nginx I get :

Problem: I am currently using ingress-nginx in my EKS cluster to route traffic to services that need public access.

My use case: I have services I want to deploy in the same cluster but don't want them to have public access. I only want the pods to communicate will all other services within the cluster. Those pods are meant to be private because they're backend services and only need pod-to-pod communication. How do I modify my ingress resource for this purpose?

Cluster Architecture: All services are in the private subnets of the cluster while the load-balancer is in the public subnets

Additional note: I am using external-dns to dynamically create the subdomains for the hosted zones. The hosted zone is public

Thanks

Below are my service.yml and ingress.yml for public services. I want to modify these files for private services

service.yml

I'm installing ingress nginx using a modified yaml file

Im trying to make an ingress for the minikube dashboard using the embedded dashboard internal service.

I enabled both ingress and dashboard minikube addons.

I also wrote this ingress YAML file :

I've googled few days and haven't found any decisions. I've tried to update k8s from 1.19.0 to 1.19.6 In Ubuntu-20. (cluster manually installed k81 - master and k82 - worker node)