kubernetes-ingress | HAProxy Kubernetes Ingress Controller | Load Balancing library

The problem you reported in your most recent comment is resolved by looking at the rewrite example in the nginx-ingress documentation.

The rewrite-target annotation configures the ingress such that matching paths will be rewritten to that value. Since you've specified a static value of /, anything matching your ingress rules will get rewritten to /, which is exactly the behavior you're seeing.

The solution is to capture the portion of the path we care about, and then use that in the rewrite-target annotation. For example:

Here are a few hints to help you out solving your issue.

Looking at the manifest files you shared, it's hard to tell which exact version of haproxy-ingress-controller container you are running in your cluster (btw, it's against best practices in production envs to leave it w/o tag, read more on it here).

For backend-server-naming configuration key to be working, minimum the v0.8.1 is required (it was backported).

Before you move on in troubleshooting, firstly please double check your ingress deployment for compatibility.

If I understand correctly the official documentation on this configuration key, setting a server naming of backends to pod names (backend-server-naming=pod) instead of sequences, does support a dynamic re-load of haproxy configuration, but does NOT support as of now dynamic updates to haproxy run-time configuration to server names at backend section (it was explained by haproxy-ingress author here, and here)

It means you need to restart your haproxy-ingress controller instance first, to be able to see changes in backend's server names reflected at haproxy configuration, e.g. situations when new Pod replicas appear or POD_IP changed due the Pod crash (expect addition/updates of server entries based on sequence naming).

I have tested successfully (see test below) the backend-server-naming=pod setting on v0.13.4 with classified Ingress type, based on ingressClassName field , rather than deprecated annotation kubernetes.io/ingress.class, as in your case:

I'm not claiming your configuration won't work (it should too), but it's important to know, that dynamic updates to configuration (this includes changes to backend configs) won't happen on unclassified Ingress resource or wrongly classified one, unless you're really running v0.12 or newer version.

This is a community wiki answer posted for better visibility. Feel free to expand it.

Based on the solution provided in the comments (method 1 example 2 in the Medium post ), a possible values.yaml file for Ingress might looks like below.

Well, I haven't figured this out for ArgoCD yet (edit: figured, but the solution is ArgoCD-specific), but for this test service it seems that path resolving is the source of the issue. It may be not the only source (to be retested on test2 subdomain), but when I created a new subdomain in the hosted zone (test3, not used anywhere before) and pointed it via A entry to the load balancer (as "alias" in AWS console), and then added to the ingress a new rule with / path, like this:

Your ClusterIssuer refers to LetsEncrypt staging issuer. Remove that setting / the default should use their production setup. As pointed out in comments: https://acme-v02.api.letsencrypt.org/directory

Deleting the previously generated secrets or switching to new secrets should ensure your certificates would be re-generated, using the right issuer.

The staging issuer could be useful testing LetsEncrypt integration, it shouldn't be used otherwise.

Your application is listening on port 8080. In your Service file you need to use the targetPort as 8080.

Seems there is a bug with the Ingress Addon with Minikube 1.23.0, as documented here, which matches the issue you are seeing. ConfigMap issues prevent IngressClass from being generated (usually "nginx" by default) and ingress services won't work.

This issue was fixed in 1.23.1, so updating Minikube should fix your issue.

Had the same issue, after days of looking for a solution, I came across metallb, from nginx ingress installation on bare metal

MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster

, from their documentation I got this

Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. The implementations of network load balancers that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.

I didn't finalize the installation but I hope the explanation above answers your question on pending status on external ip

Most of the solutions I've tried out seem to perform mTLS during the TLS handshake as nicely depicted here. So this is what I understand OSI Layer 4 (TCP/IP) authentication method.

Since TLS is above layer OSI layer 4 the authentication is also above layer 4. But OSI layers aside (which don't sufficiently match today's reality above layer 4 anyway) you essentially ask at what stage the mutual authentication happens.

Mutual authentication in TLS happens in two stages: requesting the clients certificate and validating that the certificate matches the requirements. Requesting the certificate is always done inside the TLS handshake, although it does not need to be the initial TLS handshake of the connection.

Validating the certificate can be done inside the TLS handshake, outside of it or a combination of both. Typically it is checked inside the handshake that the certificate is issued by some trusted certificate authority, but further checks for a specific subject or so might be application specific and will thus be done after the TLS handshake inside the application. But it might also be that the full validation is done inside or outside the TLS handshake.

Accepting any certificates inside the TLS handshake and validating the certificate then outside the handshake only, has the advantage that one can return a useful error message to the client inside the established TLS connection. Validation errors inside the TLS handshake instead result in cryptic errors like handshake error alerts or just closing the connection, which are not that helpful to debug the problem.