Frequently Asked Questions

Consul on Kubernetes

Q: Can I upgrade directly to a specific Helm chart version or should I upgrade one patch release at a time?

It is safe to upgrade directly to a specific version. Be sure to read the release notes for all versions you're upgrading through and look for any breaking changes.

Q: Can I upgrade in place or should I spin up a new Kubernetes cluster?

It is always safer to spin up a new Kubernetes cluster but that is not an option for most teams. Consul supports upgrading in place.

Non-production environments should be upgraded first. If upgrading a Consul version, Consul data should be backed up.

Q: How can I run tcpdump on Consul servers?

First, add the following to your values.yaml file so you can kubectl exec into the Consul server containers as root:

server:
  securityContext:
    runAsNonRoot: false
    runAsGroup: 0
    runAsUser: 0
    fsGroup: 0

Run a helm upgrade (see Upgrade Consul on Kubernetes for full upgrade instructions).

Now, kubectl exec into a server pod:

$ kubectl exec -it consul-server-0 -- sh

Install tcpdump:

$ apk add --no-cache tcpdump
$ which tcpdump
/usr/bin/tcpdump

Generic Consul Questions

Q: What is Checkpoint? / Does Consul call home?

Consul makes use of a HashiCorp service called Checkpoint which is used to check for updates and critical security bulletins. Only anonymous information, which cannot be used to identify the user or host, is sent to Checkpoint. An anonymous ID is sent which helps de-duplicate warning messages. This anonymous ID can be disabled. In fact, using the Checkpoint service is optional and can be disabled.

See disable_anonymous_signature and disable_update_check.

Q: Does Consul rely on UDP Broadcast or Multicast?

Consul uses the Serf gossip protocol which relies on TCP and UDP unicast. Broadcast and Multicast are rarely available in a multi-tenant or cloud network environment. For that reason, Consul and Serf were both designed to avoid any dependence on those capabilities.

Q: Is Consul eventually or strongly consistent?

Consul has two important subsystems, the service catalog and the gossip protocol. The service catalog stores all the nodes, service instances, health check data, ACLs, and KV information. It is strongly consistent, and replicated using the consensus protocol.

The gossip protocol is used to track which nodes are part of the cluster and to detect a node or agent failure. This information is eventually consistent by nature. When the servers detects a change in membership, or receive a health update, they update the service catalog appropriately.

Because of this split, the answer to the question is subtle. Almost all client APIs interact with the service catalog and are strongly consistent. Updates to the catalog may come via the gossip protocol which is eventually consistent meaning the current state of the catalog can lag behind until the state is reconciled.

Q: Are failed or left nodes ever removed?

To prevent an accumulation of dead nodes (nodes in either failed or left states), Consul will automatically remove dead nodes out of the catalog. This process is called reaping. This is currently done on a configurable interval of 72 hours. Reaping is similar to leaving, causing all associated services to be deregistered. Changing the reap interval for aesthetic reasons to trim the number of failed or left nodes is not advised (nodes in the failed or left state do not cause any additional burden on Consul).

Q: Does Consul support delta updates for watchers or blocking queries?

Consul does not currently support sending a delta or a change only response to a watcher or a blocking query. The API simply allows for an edge-trigger return with the full result. A client should keep the results of their last read and compute the delta client side.

By design, Consul offloads this to clients instead of attempting to support the delta calculation. This avoids expensive state maintenance on the servers as well as race conditions between data updates and watch registrations.

Q: What network ports does Consul use?

The Ports Used section of the Configuration documentation lists all ports that Consul uses.

Q: Does Consul require certain user process resource limits?

There should be only a small number of open file descriptors required for a Consul client agent. The gossip layers perform transient connections with other nodes, each connection to the client agent (such as for a blocking query) will open a connection, and there will typically be connections to one of the Consul servers. A small number of file descriptors are also required for watch handlers, health checks, log files, and so on.

For a Consul server agent, you should plan on the above requirements and an additional incoming connection from each of the nodes in the cluster. This should not be the common case, but in the worst case if there is a problem with the other servers you would expect the other client agents to all connect to a single server and so preparation for this possibility is helpful.

The default ulimits are usually sufficient for Consul, but you should closely scrutinize your own environment's specific needs and identify the root cause of any excessive resource utilization before arbitrarily increasing the limits.

Q: What is the per-key value size limitation for Consul's key/value store?

The default recommended limit on a key's value size is 512KB. This is strictly enforced and an HTTP 413 status will be returned to any client that attempts to store more than that limit in a value. The limit can be increased by using the kv_max_value_size configuration option.

It should be noted that the Consul key/value store is not designed to be used as a general purpose database. See Server Performance for more details.

Q: What data is replicated between Consul datacenters?

In general, data is not replicated between different Consul datacenters. When a request is made for a resource in another datacenter, the local Consul servers forward an RPC request to the remote Consul servers for that resource and return the results. If the remote datacenter is not available, then those resources will also not be available from that datacenter. That will not affect the requests to the local datacenter. There are some special situations where a limited subset of data can be replicated, such as with Consul's built-in ACL replication capability, or external tools like consul-replicate.

Q: Can Consul natively handle protecting against other processes accessing Consul's memory state?

Consul does not provide built-in memory access protections, and doesn't interact with the host system to change or manipulate viewing and doesn't interact with the host system to change or manipulate application security.

We recommend taking any precautions or remediation steps that you would normally do for individual processes, based on your operating system.

Please see our Security Model for more information.

Q: Are the Consul Docker Images OCI Compliant?

The official Consul Docker image uses Docker image schema V2, which is OCI Compliant. To check the docker images on Docker Hub, use the command docker manifest inspect consul to inspect the manifest payload. The docker manifest inspect may require you to enable experimental features to use.

Q: What browsers are supported by the Consul UI?

Consul currently supports all 'evergreen' browsers, as they are generally on up-to-date versions. This means we support:

• Chrome
• Firefox
• Safari
• Microsoft Edge

We do not support Internet Explorer 11 (IE 11). Consul follows a similar alignment with Microsoft's own stance on IE 11, found on their support website.