consul-replicate | Consul cross-DC KV replication daemon | Architecture library

You can use consul-replicate to keep KV data in sync between Consul data centers, but you'll want to keep a couple of things in mind about the setup.

Consul replicate uses blocking queries to watch for changes under the configured KV prefix. The way blocking queries work today is that if a single key is updated under your watched prefix, the blocking query will return the data for all keys under that prefix – even though only one key was updated – causing Consul replicate to PUT/update all watched keys in the remote DC. Depending on how many keys you are replicating, and how often those keys are updated, you may see higher bandwidth utilization and encounter performance issues.

In order to mitigate the performance concerns, you can run multiple consul-replicate processes, each one responsible for replicating a specific, scoped prefix from the KV tree (e.g., /env/prod and /env/dev) instead of the entire tree. https://github.com/hashicorp/consul/issues/2791 is a feature request to improve the behavior of watch so it only returns data for changed keys instead of all keys being watched.

Also, Consul replicate does not provide the same data guarantees as Raft within a Consul cluster. By that I mean since the replication is external to Consul, when a user/service writes to a KV in DC1, there is no way for Consul to return an error if that KV has not been successfully replicated to DC2. Consul is completely unaware the data is being replicated at all. Replication performed by consul-replicate is async and essentially best effort.

As far as setting this all up, the README for Consul replicate is fairly detailed and provides an explanation of each of the available configuration options. I imagine minimal config for a consul-replicate daemon running in DC2 which is replicating from DC1 would probably look like this.