kafka-operator | Kafka operator is a process | Pub Sub library

https://cloud.google.com/kubernetes-engine/docs/concepts/service-discovery

You might also be interesting in exposing Prometheus itself

I'm also not really sure why did you configured the advertised host and port:

So all metrics supported by Kafka Exporter are published on its metrics endpoint and when Prometheus scapes them it should scrape all of them. So if you have the other KAfka Exporter metrics in your Prometheus, you should already have this one as well (you actuall need to have some active consumer grups for it to show up).

1. Typeless API - Usage of CustomResources as raw Hashmaps
2. Typed API - Provide POJOs for CustomResource types

I'll provide examples of using both APIs to load your Kafka yaml fragment.

Typeless API:

For Typeless API you would need to provide a CustomResourceDefinitionContext, an object with details of CustomResource group, version, kind, plural etc. Here is how it would look like: KafkaLoadTypeless.java

If we are talking about a "classic" Load Balancer you would need to meet the following 2 conditions in order to use it with Kafka:

1. Load Balancers are at the TCP level ( can't use L6 or L7 Load Balancers with Kafka )
2. One Load Balancer per Kafka Broker ( just as you've mentioned, clients connect directly to the Broker that have business with )

The articles that you are mentioning are probably somehow related to Envoy Kafka Filter ( including Banzai ).

I don't know the internal details but I think I can make a summary. The main challenge with dynamic routing is the Kafka Protocol which is a TCP level protocol. So, we don't have access to metadata as we would have in the case of a higher-level protocol ( e.g: HTTP ) so that we can properly route the communication.

Envoy developed a Kafka Filter which allows just that. When a client connects to the proxy, the proxy can decode the Kafka protocol and it knows "ok, so you want to connect to x broker, let me do that for you".

There is not support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. [...] One of the distinct disadvantages of the crd-install method used in Helm 2 was the inability to properly validate charts due to changing API availability (a CRD is actually adding another available API to your Kubernetes cluster). If a chart installed a CRD, helm no longer had a valid set of API versions to work against. [...] With the new crds method of CRD installation, we now ensure that Helm has completely valid information about the current state of the cluster.

The idea here is that Helm should operate only at the level of release data (adding/removing deployments, storage, etc.); but with CRD's, you're actually modifying an extension to the Kubernetes API itself, potentially inadvertently breaking other releases that use the same definitions. Consider if you're on a team that has a "library" of CRDs shared between several charts, and you want to uninstall one — formerly, With v2, Helm would happily let you modify or even delete those at will, with no checks on if/how they were used in other releases. Changes to CRDs are changes to your control plane / core API, and should be treated as such — you're modifying global resources.

In short: with v3, Helm positions itself more as a "developer" tool to define, template, and manage releases; CRDs, however, are meant to be managed independently e.g. by a "cluster administrator". At the end of the day, it's a win for all sides, since developers can setup/teardown deployments at will, with confidence that it's not going to break functionality elsewhere... and whoever's on call won't have to deal with alerts if/when you accidentally delete/modify a CRD and break things in production :)

See also the extensive discussion here for more context behind this decision.

Hope this helps!