consul-cli | Command line interface to Consul HTTP API | REST library

Any idea, solution? I don't want to allow everyone able to create TGT, I can add service definition that only matches with CAS prefix also but first it is better to understand if I miss something or if this is a bug.

You're not missing anything. This sounds like a bug to me. As a workaround, I would add the service definition that matches the CAS prefix for now.

It sounds like this problem likely only manifests itself because you're using this:

No "client" agent is required to run for an operational Consul cluster.

I had to set this server / master with the bootstrap_expect set to 1(number of nodes for boostrap process):

The thing with consul is that at some point it wants to reach a quorum to be able to do proper leadership elections. For your servers you're using the -bootstrap-expect 2 to essentially tell the server to expect two nodes before starting the leadership election.

If you only have 2 nodes, and one is failing(or broken) you'll end up with a split brain situation. The node that is left over, doesn't have enough other nodes left to be able to decide who should be the leader inside the cluster. As a result it will not accept any new registrations.

My expectation is, that if you have a cluster of 3 nodes, and one fails, it should be able to continue running. Generally with cluster setups an un-even number of nodes is a good idea and generally (clustersize/2) >= 2

If you have a management port setup (actuators running on a separate port). You can register the management as a separate service that can be discovered.

So if you register a userservice and have set management.server.port=8081 then there will be a service registered (userservice-management) whose port is 8081 and the same host has the userservice instance.

Cause: You have a proxy or a load balancer between your consul-template and consul.

Workaround: Increase your load balancer or proxy timeout for idle connections to something above 60sec (e.g: 65sec or 120sec) because after that keepalive will kick in and will keep the connection alive :)

Details

• HTTP 504 is a gateway timeout that means the connection has been "cut" or that the service became unresponsive but I will assume that your consul cluster is still working therefore it is due to an active network device in middle (between consul-template and consul)
• Consul-template opens long lasting connections to listen for changes. Therefore if the connection is silently killed by a load balancer you end up with these 504 errors.
• These messages are benign because consul-template reconnects fine. However they are annoying and will fill your logs and disk space with the same warning over and over ;-)
• If in AWS, set your ELB idle timeout to 65sec and these messages will go away