Consul
Update Consul agents to securely communicate with TLS
Securing your datacenter with TLS encryption for RPC and consensus communication is a prerequisite of the Security Model. This tutorial provides the necessary steps to update your existing datacenter for production ready TLS with zero-downtime.
You must complete the following eight-steps successfully, completing each step before starting the next, to update agents to communicate over TLS.
Warning
Failing to perform the steps in the following order may lead to unplanned cluster outages due to failures in authenticating RPC TLS Communication. Complete each step in its entirety prior to moving to the next one.
- Generate server certificates. Generate a client certificate if you're using the operator method.
- Distribute certificates, including the CA and private key files to every agent.
- Configure the servers, allowing encrypted traffic.
- Restart each server, one at a time.
- Configure the clients to communicate only with TLS
- Restart each client.
- Reconfigure servers to only allow TLS communication.
- Reload each server, one at a time.
If you want to secure service-to-service communication with TLS, review the secure service communication tutorial.
Prerequisites
Before starting this tutorial, you should have an existing Consul datacenter with running server and client agents.
You will need to generate your certificates before starting this tutorial. Follow the steps in the Secure Agent Communication with TLS Encryption tutorial to use the built-in CA or the Secure Agent Communication with Existing Certificate Authority tutorial to use OpenSSL.
Finally, you should decide which certificate distribution method works for you. There are two methods for distributing client certificates; operator or auto encryption. Both methods are described in the "Secure Agent Communication with TLS Encryption" tutorial.
The example configurations for this tutorial are in JSON. You can copy each one of the examples and add them to your existing JSON configuration file.
Distribute the certificates
Before configuring the agents, you will need to distribute the certificate and key files.
Server certificates
Distribute the following certificates and key files to each of your servers.
Note, the CA certificate may be named differently if you are not using Consul's built-in CA. The server certificate and key file should be unique for each server.
Client certificate for operator method
Distribute the client certificate and key file to all your clients. All clients should have the following certificates and key file.
Configure the servers
Now that you have created the certificates you can enable TLS on your agents, starting with the servers. For both client certificate distribution methods, operator and auto encryption, you must set the following options to false.
These options must be set to false to allow existing communication to continue uninterrupted. This will require a rolling restart on all the servers. Once the clients are configured, you can then enable verify_incoming, verify_outgoing, and verify_server_hostname on the servers. In addition to the verify options, you will set the certificates and key file the agent should use.
Below are the configurations for both methods.
Note
Consul will load all files in the configuration directory, for the following examples, you can add the configuration
as new files. You can also add them to an existing configuration file, however, you should ensure that the syntax is valid before applying
it to the agent. You can use consul validate to check configuration validity for
both JSON and HCL files.
Operator method configuration
Consul server agent manual TLS configuration
verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
Auto encryption method configuration
For the auto encryption method, you will need to set auto_encrypt to true.
Consul server agent auto encrypt TLS configuration
verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
auto_encrypt {
allow_tls = true
}
Setting the verify options to false on the servers allows you to
update all the agents without any downtime since authenticity verification is
not enforced.
Note
Leaving the verify settings as false will create an
insecure configuration.
Restart the Consul servers
After configuring the servers, for either the operator or auto encryption client certificate distribution method, you can initiate a rolling restart of the servers.
Be sure to restart one server at a time, ensuring it's functioning properly before restarting the next. The leader should be restarted last, to avoid downtime.
$ systemctl restart consul
Configure the clients
Now that you have enabled TLS communication on the servers and distributed the certificates, you can configure the clients.
Operator method
For the operator method, you will configure the clients to use the operator distributed certificates and key file.
Consul client agent manual TLS configuration
verify_incoming = false
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-client-consul-0.pem"
key_file = "dc1-client-consul-0-key.pem"
Auto encryption method
For the auto encryption method, you will enable auto_encrypt.
Consul client agent auto encrypt TLS configuration
verify_incoming = false
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
auto_encrypt = {
tls = true
}
You do not need to distribute client certificate using the auto encryption method. Consul service mesh will manage certificate distribution for you.
Restart the Consul clients
Finally, restart the clients. You will only need to restart the clients once.
$ systemctl restart consul
Warning
Restarting your Consul server agents with the consul reload command may lead to an outage since this will cause the clients to lose communication to cluster server nodes. To prevent this, adjust the TLS verification settings as prescribed in the following steps.
Reconfigure and reload the servers
Operator method
For the operator method, you will update verify_incoming,
verify_outgoing, and verify_server_hostname to true.
Consul server agent manual TLS configuration
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
Auto encryption method
For the auto encryption method, you will update verify_incoming,
verify_outgoing, and verify_server_hostname to true
Consul server agent auto encrypt configuration
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
auto_encrypt {
allow_tls = true
}
Reload the Consul servers
After a final Consul agent reload, your agents will only communicating with TLS.
$ consul reload
Next steps
In this tutorial, you configured your Consul agents to communicate over TLS. The other prerequisites for a secure Consul deployment are gossip encryption and ACLs with the agents configured with default deny.
Additional TLS encryption resources
For more TLS content review our other tutorials:
Generate TLS certificates with OpenSSL and secure the CLI and UI communication.