haproxy-ingress | HAProxy Ingress | Load Balancing library

Managed Certificates only work with L7 (HTTP) LoadBalancer not with TCP ones.

My understanding is you want to use nginx as an Ingress controller on GKE but you want to expose it behind an L7 LoadBalancer so you can use Google Managed Certificates ?

Here are a few hints to help you out solving your issue.

Looking at the manifest files you shared, it's hard to tell which exact version of haproxy-ingress-controller container you are running in your cluster (btw, it's against best practices in production envs to leave it w/o tag, read more on it here).

For backend-server-naming configuration key to be working, minimum the v0.8.1 is required (it was backported).

Before you move on in troubleshooting, firstly please double check your ingress deployment for compatibility.

If I understand correctly the official documentation on this configuration key, setting a server naming of backends to pod names (backend-server-naming=pod) instead of sequences, does support a dynamic re-load of haproxy configuration, but does NOT support as of now dynamic updates to haproxy run-time configuration to server names at backend section (it was explained by haproxy-ingress author here, and here)

It means you need to restart your haproxy-ingress controller instance first, to be able to see changes in backend's server names reflected at haproxy configuration, e.g. situations when new Pod replicas appear or POD_IP changed due the Pod crash (expect addition/updates of server entries based on sequence naming).

I have tested successfully (see test below) the backend-server-naming=pod setting on v0.13.4 with classified Ingress type, based on ingressClassName field , rather than deprecated annotation kubernetes.io/ingress.class, as in your case:

I'm not claiming your configuration won't work (it should too), but it's important to know, that dynamic updates to configuration (this includes changes to backend configs) won't happen on unclassified Ingress resource or wrongly classified one, unless you're really running v0.12 or newer version.

This issue is due to a bug in https://github.com/haproxytech/kubernetes-ingress when using IngressClassName in ingress.yaml. If you remove IngressClassName from your ingress.yaml and just use "kubernetes.io/ingress.class": annotation the issue goes away, it more of a workaround than a fix.

This issue has been raised and still open see link below for updates.

https://github.com/haproxytech/kubernetes-ingress/issues/354#issuecomment-904551220

Unfortunately without logs it's hard to say anything for sure. You should check the audit logs that GKE ships to Cloud Logging as that might give you some idea of what happened. One option is the GCP "oops"'d the GLB and GKE recreated it, thus giving it a new IP. I've never heard of that happening with LBs though (it happens pretty often with nodes, but not LBs). A more common case would be you ran some kubectl command that inadvertently removed the Service object and then it was recreated by some management layer you have set up (Argo, Flux, Helm Operator, whatever) but delete+recreate again means it's a new LB with a new IP. The latter case should be visible in the audit logs so check those out for sure.

Yes it is possible.

There can be two issues in your case.

One is you don't need the regex path for grafana ingress. Simple /grafana path will be fine with path type Prefix as with path type Prefix any /grafana/... will be redirected associated service. So the manifest file will be:

I finally figured out what I was missing. I added kubernetes.io/ingress.class: haproxy annotation and problem solved:

Thanks to Mariusz K. I found the answer to my problem. In case that someone else might have the same problem here is how I solved it.

First query the cluster (HA) for the etcd members (sample of code):

I got the answer. From the document of helmfile,

You can use go's text/template expressions in helmfile.yaml and values.yaml.gotmpl (templated helm values files). values.yaml references will be used verbatim. In other words: for value files ending with .gotmpl, template expressions will be rendered for plain value files (ending in .yaml), content will be used as-is

Therefore, for this case, values.yaml should be renamed to values.yaml.gotmpl so that helmfile will interpret the file as go template.

HAProxy by default allows to reuse the same port number across the same or other frontend/listen sections and also across other haproxy process. This can be changed adding noreuseport in the global section.

The default HAProxy Ingress configuration uses port number 1936 to expose stats. If such port number is reused by eg a tcp proxy, the incoming requests will be distributed between both frontends - sometimes your service will be called, sometimes the stats page. Changing the tcp proxy or the stats page (doc here) to another port should solve the issue.