marathon-api | Easy to use Marathon API written in ruby | REST library

You'll need to use a "real" downing provider like Split Brain Resolver. This lets the cluster safely down nodes that are unreachable. (As opposed to the auto downing, which downs them without consideration of it is safe or not.)

There's a separate question of why DC/OS is killing the nodes so quickly they don't get the chance to properly shut down. But I don't know DC/OS well enough to say why that could be. And, regardless, a downing provider is essential for clustered environments so you will want to get that in place anyway.

Edited due to your comments about SBR:

• First I want to be clear, the marathan-api is almost certainly irrelevant here. marathon-api is how nodes discover other nodes in DC/OS. The problems you are having are with fundamental cluster problems, namely unreachable nodes. A cluster with unreachable nodes is going to act the same way, regardless of where it is running and how the nodes are discovered.
• Fundamentally my best guess is that you are having problems getting clean shutdowns. If SBR is downing your entire cluster it's because it is getting to a point where there are more unreachable nodes than live clusters.

As an example, what might be happening:

• You have 4 live nodes and want to upgrade.
• DC/OS kills the first node. For some unknown reason you aren't getting a clean shutdown so the node is marked as unreachable. (Essentially the cluster, because it wasn't a clean shutdown doesn't node if the node still exists but is unresponsive and/or behind a partition.) There are 3 live nodes and one unreachable.
• DC/OS starts the second node. Perhaps it takes a while to boot your application. So you have 3 live nodes, one unreachable node, and 1 unready node.
• DC/OS kills another node. So you have 2 live nodes, 2 unreachable nodes, and 1 unready node. At this point SBR can no longer guarantee that you haven't had a network partition because you don't have majority. At this point, in order to prevent corruption, it must stop the cluster.

So, I would recommend the following:

• I don't know the details of DC/OS well enough but you probably need to slow down your rolling upgrades. In K8S I'd use something like MinReadySeconds.
• You may want to consider a fifth node. Odd numbers are better because it makes majority easier to determine.
• If you continue to have problems you'll need to provide more logs from the SBR decision.

SBR is the answer here. I realize that you aren't having real network partitions, but the fact that you are having unreachable nodes means that the Akka Cluster is unable to tell if there are network partitions or not and that's the root cause of the problem.