q-cluster | Blends client | Machine Learning library

an operator pod acts as a controller that listens to events regarding specific custom resources. if you only deploy the operator, you have to seperately deploy the custom resource you wish to be created.

with the prometeus-operator, that would be a custom resource of kind "prometheus". if the helm chart you choose is capable to also deploy this (or not) should be indicated in the charts values.yaml and documented on their github page.

you can also use the examples from the prometheus-operator repo to create prometheus instances. check out these files to do so: https://github.com/prometheus-operator/prometheus-operator/tree/main/example/rbac/prometheus

The command docker rm -f (forcibly) removes a container. You cannot start it anymore after that, because it no longer exists.

If you want to create a new container from the service rabbit_node_3 in your docker-compose file, you need to use the docker-compose command from within the directory where your docker-compose.yml resides:

docker-compose up -d rabbit_node_3

Where:

• up tells docker-compose to create containers from the named service (rabbit_node_3)
• -d tells docker-compose that you do not want to attach your terminal to stdout and stderr of the container started with this command. It would just go in the background and free your terminal for other tasks

The issue from the error message seems to be that RabbitMQ thinks you are providing it an advanced configuration file instead of the normal configuration file - https://www.rabbitmq.com/configure.html#advanced-config-file . Even though since RabbitMQ 3.7+ has sysctl(the format you used) kind of configuration files, the advanced configuration file still uses the classic configuration format(https://www.rabbitmq.com/configure.html#config-file-formats) which explains the syntax error.

From the docs - https://www.rabbitmq.com/configure.html#configuration-files

Not sure why it would pick the value of the RABBITMQ_CONFIG_FILE as the advanced config file instead of the default one.

Can you update the question with the full logs? Even after the container is dead, you can check its logs using docker logs rabbit-1

You can use port forwarding the rabbitMQ service to your local machine and use UI login and check the password with the UI given by the RabbitMQ itself.

According to the command you used to start minikube, the error is because you don't have enough resources assigned to your cluster.
According to the source code from the rabbitmq cluster operator, it seems that it needs 2CPUs.

You need to adjust the number of CPUs (and probably the memory also) when you initialize your cluster. Below is an example to start a kubernetes cluster with 4 cpus and 8G of RAM :

From my understanding the connectors will be used when forming the master slave pairs and then the replication will be done via SSL using the configuration from connectors section is this the case?

Yes, that is the case.

I believe the issue here was that helm was configuring so much of my own stuff that I wasn't able to customise too much.

I uninstalled the ingress with helm and changed the ingress creation script to this:

I solved it by following the logs itself

it clearly says "CLI tools must refer to node "rabbit@ip-10-0-x-xxx" using the same name the node itself uses"

while passing the cluster node name i was doing it wrong NOTE : added hostname from cat /etc/hosts

did like below sudo rabbitmqctl join_cluster rabbit@ip-10-0-x-yyy

and it worked suucessfull message "Clustering node rabbit@ip-10-0-x-xxx with rabbit@ip-10-0-x-yyy"

So thanks to @NeoAnderson and @José M, I was able to understand what happened.

The containers running RMQ need to be accessible via the hostname that Erlang uses within the service, across the network. Since the hostname of the containers were not accessible in a container on another machine, this clustering failed.

A simple fix would be to edit the /etc/hosts file on the containers so that it would point the IP to the "leader" node.

I was just doing this to avoid installing RMQ and not because I thought this was the best way to do this. Alternately, docker swarm or k8s would have provided the right networking for me.

But the root cause was definitely the nodename problem.