nginx-cluster | A horizontally scalable NGINX caching cluster | Caching library

If I understood you correctly, there are a few ways you might be looking at.

One would be MetalLB which you already mentioned.

MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.

You can read the detailed implementation A pure software solution: MetalLB

Another way would be Over a NodePort Service

This approach has a few other limitations one ought to be aware of:

• Source IP address

Services of type NodePort perform source address translation by default. This means the source IP of a HTTP request is always the IP address of the Kubernetes node that received the requestfrom the perspective of NGINX.

You can also use host network

In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure ingress-nginx Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of this approach is that the NGINX Ingress controller can bind ports 80 and 443 directly to Kubernetes nodes' network interfaces, without the extra network translation imposed by NodePort Services.

You have to also remember that if you edit the configuration inside the POD, it will be gone if the Pod is restarted or it crashes.

I hope this helps you to determine which way to go with your idea.