HashiConf 2024 Now streaming live from Boston! Attend for free
Consul
Consul Home

Fault tolerance

You must give careful consideration to reliability in the architecture frameworks that you build. When you build a resilient platform, it minimizes the remediation actions you need to take when a failure occurs. This document provides useful information on how to design and operate a resilient Consul cluster, including the methods and functionalities for this goal.

Consul has many features that operate both locally and remotely that can help you offer a resilient service across multiple datacenters.

Introduction

Fault tolerance is the ability of a system to continue operating without interruption despite the failure of one or more components. In Consul, the number of server agents determines the fault tolerance.

Each Consul datacenter depends on a set of Consul voting server agents. The voting servers ensure Consul has a consistent, fault-tolerant state by requiring a majority of voting servers, known as a quorum, to agree upon any state changes. Examples of state changes include: adding or removing services, adding or removing nodes, and changes in service or node health status.

Without a quorum, Consul experiences an outage: it cannot provide most of its capabilities because they rely on the availability of this state information. If Consul has an outage, normal operation can be restored by following the Disaster recovery for Consul clusters guide.

If Consul is deployed with 3 servers, the quorum size is 2. The deployment can lose 1 server and still maintain quorum, so it has a fault tolerance of 1. If Consul is instead deployed with 5 servers, the quorum size increases to 3, so the fault tolerance increases to 2. To learn more about the relationship between the number of servers, quorum, and fault tolerance, refer to the consensus protocol documentation.

Effectively mitigating your risk is more nuanced than just increasing the fault tolerance because the infrastructure costs can outweigh the improved resiliency. You must also consider correlated risks at the infrastructure-level. There are occasions when multiple servers fail at the same time. That means that a single failure could cause a Consul outage, even if your server-level fault tolerance is 2.

Different options for your resilient datacenter present trade-offs between operational complexity, computing cost, and Consul request performance. Consider these factors when designing your resilient architecture.

Fault tolerance

The following sections explore several options for increasing Consul's fault tolerance. For enhanced reliability, we recommend taking a holistic approach by layering these multiple functionalities together.

Availability zones

The cloud or on-premise infrastructure underlying your Consul datacenter can run across multiple availability zones.

An availability zone is meant to share no points of failure with other zones by:

  • Having power, cooling, and networking systems independent from other zones
  • Being physically distant enough from other zones so that large-scale disruptions such as natural disasters (flooding, earthquakes) are very unlikely to affect multiple zones

Availability zones are available in the regions of most cloud providers and in some on-premise installations. If possible, spread your Consul voting servers across 3 availability zones to protect your Consul datacenter from a single zone-level failure. For example, if deploying 5 Consul servers across 3 availability zones, place no more than 2 servers in each zone. If one zone fails, at most 2 servers are lost and quorum will be maintained by the 3 remaining servers.

To distribute your Consul servers across availability zones, modify your infrastructure configuration with your infrastructure provider. No change is needed to your Consul server's agent configuration.

Additionally, you should leverage resources that can automatically restore your compute instance, such as autoscaling groups, virtual machine scale sets, or compute engine autoscaler. Customize autoscaling resources to re-deploy servers into specific availability zones and ensure the desired numbers of servers are available at all times.

Quorum size

For most production use cases, we recommend using a minimum quorum of either 3 or 5 voting servers, yielding a server-level fault tolerance of 1 or 2 respectively.

Even though it would improve fault tolerance, adding voting servers beyond 5 is not recommended because it decreases Consul's performance— it requires Consul to involve more servers in every state change or consistent read.

Consul Enterprise users can use redundancy zones to improve fault tolerance without this performance penalty.

Redundancy zones Enterprise

Use Consul Enterprise redundancy zones to improve fault tolerance without the performance penalty of increasing the number of voting servers.

Reference architecture diagram for Consul Redundancy zones Reference architecture diagram for Consul Redundancy zones

Each redundancy zone should be assigned 2 or more Consul servers. If all servers are healthy, only one server per redundancy zone will be an active voter; all other servers will be backup voters. If a zone's voter is lost, it will be replaced by:

  • A backup voter within the same zone, if any. Otherwise,
  • A backup voter within another zone, if any.

Consul can replace lost voters with backup voters within 30 seconds in most cases. Because this replacement process is not instantaneous, redundancy zones do not improve immediate fault tolerance— the number of healthy voting servers that can fail at once without causing an outage. Instead, redundancy zones improve optimistic fault tolerance: the number of healthy active and back-up voting servers that can fail gradually without causing an outage.

The relationship between these two types of fault tolerance is:

Optimistic fault tolerance = immediate fault tolerance + the number of healthy backup voters

For example, consider a Consul datacenter with 3 redundancy zones and 2 servers per zone. There will be 3 voting servers (1 per zone), meaning a quorum size of 2 and an immediate fault tolerance of 1. There will also be 3 backup voters (1 per zone), each of which increase the optimistic fault tolerance. Therefore, the optimistic fault tolerance is 4. This provides performance similar to a 3 server setup with fault tolerance similar to a 7 server setup.

We recommend associating each Consul redundancy zone with an infrastructure availability zone to also gain the infrastructure-level fault tolerance benefits provided by availability zones. However, Consul redundancy zones can be used even without the backing of infrastructure availability zones.

For more information on redundancy zones, refer to:

Autopilot

Autopilot is a set of functions that introduce servers to a cluster, cleans up dead servers, and monitors the state of the Raft protocol in the Consul cluster.

When you enable Autopilot's dead server cleanup, Autopilot marks failed servers as Left and removes them from the Raft peer set to prevent them from interfering with the quorum size. Autopilot does that as soon as a replacement Consul server comes online. This behavior is beneficial when server nodes failed and have been redeployed but Consul considers them as new nodes because their IP address and hostnames have changed. Autopilot keeps the cluster peer set size correct and the quorum requirement simple.

To illustrate the Autopilot advantage, consider a scenario where Consul has a cluster of five server nodes. The quorum is three, which means the cluster can lose two server nodes before the cluster fails. The following events happen:

  1. Two server nodes fail.
  2. Two replacement nodes are deployed with new hostnames and IPs.
  3. The two replacement nodes rejoin the Consul cluster.
  4. Consul treats the replacement nodes as extra nodes, unrelated to the previously failed nodes.

With Autopilot not enabled, the following happens:

  1. Consul does not immediately clean up the failed nodes when the replacement nodes join the cluster.
  2. The cluster now has the three surviving nodes, the two failed nodes, and the two replacement nodes, for a total of seven nodes.
    • The quorum is increased to four, which means the cluster can only afford to lose one node until after the two failed nodes are deleted in seventy-two hours.
    • The redundancy level has decreased from its initial state.

With Autopilot enabled, the following happens:

  1. Consul immediately cleans up the failed nodes when the replacement nodes join the cluster.
  2. The cluster now has the three surviving nodes and the two replacement nodes, for a total of five nodes.
    • The quorum stays at three, which means the cluster can afford to lose two nodes before it fails.
    • The redundancy level remains the same.

Cluster peering

Linking multiple Consul clusters together to provide service redundancy is the most effective method to prevent disruption from failure. This method is enhanced when you design individual Consul clusters with resilience in mind. Consul clusters interconnect in two ways: WAN federation and cluster peering. We recommend using cluster peering whenever possible.

Cluster peering lets you connect two or more independent Consul clusters using mesh gateways, so that services can communicate between non-identical partitions in different datacenters.

Reference architecture diagram for Consul cluster peering Reference architecture diagram for Consul cluster peering

Cluster peering is the preferred way to interconnect clusters because it is operationally easier to configure and manage than WAN federation. Cluster peering communication between two datacenters runs only on one port on the related Consul mesh gateway, which makes it operationally easy to expose for routing purposes.

When you use cluster peering to connect admin partitions between datacenters, use Consul’s dynamic traffic management functionalities service-splitter, service-router and service-failover to configure your service mesh to automatically forward or failover service traffic between peer clusters. Consul can then manage the traffic intended for the service and do failover, load-balancing, or redirection.

Cluster peering also extends service discovery across different datacenters independent of service mesh functions. After you peer datacenters, you can refer to services between datacenters with <service>.virtual.peer.consul in Consul DNS. For Consul Enterprise, your query string may need to include the namespace, partition, or both. Refer to the Consul DNS documentation for details on building virtual service lookups.

For more information on cluster peering, refer to:

Edit this page on GitHub