Generate mTLS Certificates for Consul with Vault

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates with consul-template to rotate your certificates. This method enables each agent in the Consul datacenter to have a unique certificate with a relatively short time-to-live (ttl) that is automatically rotated, which allows you to safely and securely scale your datacenter while using mutual TLS (mTLS).

In this tutorial, your will secure your Consul datacenter with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use consul-template to fetch, renew, and periodically rotate your mTLS certificates on your Consul nodes.

Prerequisites

• Consul: to complete this tutorial, you can use a local Consul dev agent or an existing Consul deployment.
• Vault: the tutorial assumes you already have a running Vault cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.
• consul-template: to interact with your Consul agent you will need to install the consul-template binary on the node. To rotate TLS certificates you need the binary to be installed on every node in the datacenter. You can check Service Configuration with Consul Template to learn how to install and use consul-template.

The diagram below shows the minimal architecture needed to demonstrate the functionality.

Configure Vault's PKI secrets engine

In order to communicate with the Vault server, you will need to set the address and token.

$ export VAULT_ADDR='http://127.0.0.1:8200'

$ export VAULT_TOKEN="root"

Enable Vault's PKI secrets engine at the pki path.

$ vault secrets enable pki

Success! Enabled the pki secrets engine at: pki/

Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.

$ vault secrets tune -max-lease-ttl=87600h pki

Success! Tuned the secrets engine at: pki/

Configure Vault as Consul's CA

Consul requires that all servers and clients have key pairs that are generated by a single Certificate Authority (CA).

In this lab you will use the PKI secrets engine to generate the necessary CA and certificates.

1. Generate the root CA

Generate the root certificate and save the certificate in CA_cert.crt.

$ vault write -field=certificate pki/root/generate/internal \
        common_name="dc1.consul" \
        ttl=87600h > CA_cert.crt

This generates a new self-signed CA certificate and private key. Vault will automatically revoke the generated root at the end of its lease period (TTL); the CA certificate will sign its own Certificate Revocation List (CRL).

Configure the CA and CRL URLs.

$ vault write pki/config/urls \
        issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
        crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"

Example output:

Success! Data written to: pki/config/urls

2. Generate an intermediate CA

First, enable the PKI secrets engine at the pki_int path.

$ vault secrets enable -path=pki_int pki

Success! Enabled the pki secrets engine at: pki_int/

Tune the pki_int secrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours.

$ vault secrets tune -max-lease-ttl=43800h pki_int

Success! Tuned the secrets engine at: pki_int/

Request an intermediate certificate signing request (CSR) and save request as pki_intermediate.csr.

$ vault write -format=json pki_int/intermediate/generate/internal \
        common_name="dc1.consul Intermediate Authority" \
        | jq -r '.data.csr' > pki_intermediate.csr

The command has no output.

3. Sign the CSR and import the certificate into Vault

$ vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr \
        format=pem_bundle ttl="43800h" \
        | jq -r '.data.certificate' > intermediate.cert.pem

The command has no output.

Once the CSR is signed, and the root CA returns a certificate, it can be imported back into Vault.

$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Success! Data written to: pki_int/intermediate/set-signed

4. Create a Vault role

A role is a logical name that maps to a policy used to generate credentials.

$ vault write pki_int/roles/consul-dc1 \
  allowed_domains="dc1.consul" \
  allow_subdomains=true \
  generate_lease=true \
  max_ttl="720h"

Example output:

Success! Data written to: pki_int/roles/consul-dc1

For this tutorial, you are using the following options for the role:

• allowed_domains : Specifies the domains of the role. The command uses dc1.consul as the domain, which is the default configuration you are going to use for Consul.
• allow_subdomains : Specifies if clients can request certificates with CNs that are subdomains of the CNs allowed by the other role options (NOTE: This includes wildcard subdomains.)
• generate_lease : Specifies if certificates issued/signed against this role will have Vault leases attached to them. Certificates can be added to the CRL by Vault revoke <lease_id> when certificates are associated with leases.

This completes the Vault configuration as a CA. Move to next step to generate certificates.

Generate a server certificate

You can test the pki engine is configured correctly by generating your first certificate.

$ vault write pki_int/issue/consul-dc1 \
  common_name="server.dc1.consul" \
  ttl="24h" | tee certs.txt

Example output:

Key                 Value
---                 -----
lease_id            pki_int/issue/consul-dc1/lFfKfpxtM0xY0AHDlr9pJ2GM
lease_duration      23h59m59s
lease_renewable     false
ca_chain            [-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----]
certificate         -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
expiration          1599645187
issuing_ca          -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       3f:ec:bd:ea:01:a6:35:49:a7:6d:17:ba:13:88:c1:b8:35:b4:fc:4c

Configure Consul

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/opt/consul/agent-certs/ca.crt"
cert_file = "/opt/consul/agent-certs/agent.crt"
key_file = "/opt/consul/agent-certs/agent.key"
auto_encrypt {
  allow_tls = true
}

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "/opt/consul/agent-certs/ca.crt",
  "cert_file": "/opt/consul/agent-certs/agent.crt",
  "key_file": "/opt/consul/agent-certs/agent.key",
  "auto_encrypt": {
    "allow_tls": true
  }
}

$ mkdir -p /opt/consul/agent-certs

$ grep -Pzo "(?s)(?<=certificate)[^\-]*.*?END CERTIFICATE[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/agent.crt

$ grep -Pzo "(?s)(?<=issuing_ca)[^\-]*.*?END CERTIFICATE[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/ca.crt

$ grep -Pzo "(?s)(?<=private_key)[^\-]*.*?END RSA PRIVATE KEY[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/agent.key

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/opt/consul/agent-certs/ca.crt"
auto_encrypt {
  tls = true
}

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "/opt/consul/agent-certs/ca.crt",
  "auto_encrypt": {
    "tls": true
  }
}

$ mkdir -p /opt/consul/agent-certs

$ grep -Pzo "(?s)(?<=certificate)[^\-]*.*?END CERTIFICATE[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/agent.crt

Configure consul-template

When you created the certificate you used as a parameter ttl="24h" meaning that this certificates will be valid only for 24 hours before expiring.

Deciding the right trade-off for certificate lifespan is always a compromise between security and agility. A possible third way that does not require you to lower your security is to use consul-template to automate certificate renewal for Consul when the TTL is expired.

Create template files

You can instruct consul-template to generate and retrieve those files from Vault using the following templates:

{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.certificate }}
{{ end }}

The template will use the pki_int/issue/consul-dc1 endpoint that Vault exposes to generate new certificates. It also mentions the common name and alternate names for the certificate.

{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.private_key }}
{{ end }}

{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h"}}
{{ .Data.issuing_ca }}
{{ end }}

Once you have created the templates for consul-template, you can copy the files into /opt/consul/templates.

$ cp *.tpl /opt/consul/templates/

Create consul-template configuration

For this tutorial, you are going to create a configuration file, consul_template.hcl, that will instruct consul-template to retrieve the files needed for the agent, client and server, to configure TLS encryption.

# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "http://localhost:8200"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  token        = "root"

  unwrap_token = false

  renew_token  = false
}

# This block defines the configuration for a template. Unlike other blocks,
# this block may be specified multiple times to configure multiple templates.
template {
  # This is the source file on disk to use as the input template. This is often
  # called the "consul-template template".
  source      = "agent.crt.tpl"

  # This is the destination path on disk where the source template will render.
  # If the parent directories do not exist, consul-template will attempt to
  # create them, unless create_dest_dirs is false.
  destination = "/opt/consul/agent-certs/agent.crt"

  # This is the permission to render the file. If this option is left
  # unspecified, consul-template will attempt to match the permissions of the
  # file that already exists at the destination path. If no file exists at that
  # path, the permissions are 0644.
  perms       = 0700

  # This is the optional command to run when the template is rendered. The
  # command will only run if the resulting template changes.
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "agent.key.tpl"
  destination = "/opt/consul/agent-certs/agent.key"
  perms       = 0700
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "ca.crt.tpl"
  destination = "/opt/consul/agent-certs/ca.crt"
  command     = "sh -c 'date && consul reload'"
}

# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "http://localhost:8200"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  token        = "root"

  unwrap_token = false

  renew_token  = false
}

template {
  source      = "ca.crt.tpl"
  destination = "/opt/consul/agent-certs/ca.crt"
  command     = "sh -c 'date && consul reload'"
}

The following parameters are needed in order to allow consul-template to communicate with Vault:

• address : the address of your Vault server. In this tutorial, Vault runs on the same node as Consul so you can use http://localhost:8200.

• token : a valid Vault ACL token with appropriate permissions. You will use Vault root token for this tutorial.

Start consul-template

After configuration is completed, you can start consul-template. You must provide the file with the -config parameter.

$ consul-template -config "consul_template.hcl"

Configuration reload triggered

Verify certificate rotation

The certificate you created manually for the Consul server had a TTL of 24 hours.

This means that after the certificate expires Vault will renew it and consul-template will update the files on your agent it reload Consul configuration automatically to make it pick up the new files.

You can verify the rotation by checking that consul-template keeps listing, every 24 hours, a timestamp and the log line:

Configuration reload triggered

You can also use openssl to verify the certificate content:

$ openssl x509 -text -noout -in /opt/consul/agent-certs/agent.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1b:2d:d6:5d:63:9b:aa:05:84:7b:be:3b:6f:e1:95:bb:1c:36:8c:a4
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=dc1.consul Intermediate Authority
        Validity
            Not Before: Sep 16 16:03:45 2020 GMT
            Not After : Sep 16 16:06:15 2020 GMT
        Subject: CN=server.dc1.consul
...

and verify that the Not Before and Not After values are being updated to reflect the new certificate.

Next steps

In this tutorial you learned how to configure Consul TLS encryption using Vault's PKI secrets engine to generate short lived certificates and using consul-template to automatically rotate them at the end of their lease time.

To learn how to configure gossip encryption for Consul using Vault as a secret storage management you can follow Secure Consul Gossip Communication with Vault tutorial.

To learn how to create Consul tokens using Vault you can follow Generate Consul Tokens with HashiCorp Vault tutorial.