Vault as Consul service mesh certification authority

30min
|
Enterprise
Consul
Vault
Video
Interactive

You can use Vault's PKI Secrets Engine to generate and renew dynamic TLS X.509 certificates. Aside from being useful to secure Consul's RPC and Raft communication, the PKI secrets engine provides support serving as a Certificate Authority for Consul service mesh.

One of the benefits of using a service mesh is creating, generating, and rotating TLS certificates for services in the service mesh. Using Vault as the CA for your service mesh provides unified management for TLS certificates.

This tutorial configures a Consul datacenter to use Vault as the Consul service mesh CA for new and existing Consul datacenters.

Prerequisites

Click here for an example configuration

To test the commands provided by this tutorial you need a Consul datacenter configured with ACL, TLS encryption, and Gossip encryption enabled. You can use the following configuration file as an example to set up a single node Consul datacenter.

# Datacenter configuration
domain = "consul"
datacenter = "dc1"
node_name = "consul-server-0"

data_dir = "/<absolute-path-to>/consul/data"

# Server configuration
server = true
bootstrap_expect = 1

ui_config {
  enabled = true
}

ports {
  grpc = 8502
}

# TLS Encryption configuration
verify_incoming        = false
verify_incoming_rpc    = true
verify_outgoing        = true
verify_server_hostname = true

ca_file   = "/<absolute-path-to>/consul-agent-ca.pem"
cert_file = "/<absolute-path-to>/dc1-server-consul-0.pem"
key_file  = "/<absolute-path-to>/dc1-server-consul-0-key.pem"

auto_encrypt = {
    allow_tls = true
}

# ACL configuration
# After startup, bootstrap the ACL system with `consul acl bootstrap` command
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

# Gossip Encryption
encrypt = "tRgXSb6ClvzV1myhc75rWIdwx8tTmUI8UxySKecxiQA="

# If running Consul 1.8.x or below, enable central service configuration
enable_central_service_config = true

# You can Append the Connect configuration here or place it in a separate
# configuration file.
# ...

More details on how to securely configure your consul datacenter are available in the following tutorials:

Vault: the tutorial assumes you already have a running Vault 0.10.3+ cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Security Warning

This tutorial is not for production use. By default, the lab will install an insecure configuration of Consul.

Configure Vault ACL

To interact with the PKI Secrets Engine endpoints, you have to generate a Vault token, giving the appropriate permissions to Consul to generate the certificates.

Create Vault policies

The following Vault policies apply to Consul 1.12 and later. For use with earlier Consul versions, refer to the Vault CA provider documentation and select your version from the version dropdown.

If you want to manually create and tune the PKI secret engines to be used as CA, you can create them in Vault before enabling the integration.

In this example, the RootPKIPath is connect_root and the IntermediatePKIPath is connect_dc1_inter. Update these values to reflect your environment.

First, enable the PKI secrets engine for the root CA.

$ vault secrets enable -path=connect_root pki

Success! Enabled the pki secrets engine at: connect_root/

Next, enable the PKI secrets engine for the intermediate CA.

$ vault secrets enable -path=connect_dc1_inter pki

Success! Enabled the pki secrets engine at: connect_dc1_inter/

Finally, create a policy in Vault for the integration.

The following Vault policy allows Consul to use pre-existing PKI paths in Vault. Consul is granted read-only access to the PKI mount points and the root CA, but is granted full control of the intermediate or leaf CA for Consul service mesh clients.

path "/sys/mounts/connect_root" {
  capabilities = [ "read" ]
}

path "/sys/mounts/connect_dc1_inter" {
  capabilities = [ "read" ]
}

path "/sys/mounts/connect_dc1_inter/tune" {
  capabilities = [ "update" ]
}

path "/connect_root/" {
  capabilities = [ "read" ]
}

path "/connect_root/root/sign-intermediate" {
  capabilities = [ "update" ]
}

path "/connect_dc1_inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "auth/token/renew-self" {
  capabilities = [ "update" ]
}

path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}

The following Vault policy allows Consul to create and manage the PKI paths in Vault. Consul is granted the ability to create the PKI mount points and given full control of the root and intermediate CA and to generate leaf certificates for Consul service mesh clients.

The leaf certificates are created for each service instance and are used to verify the identity of the services and to secure communication across service instances.

In this example, the RootPKIPath is connect_root and the IntermediatePKIPath is connect_dc1_inter. Update these values to reflect your environment.

path "/sys/mounts/connect_root" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect_dc1_inter" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect_dc1_inter/tune" {
  capabilities = [ "update" ]
}

path "/connect_root/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_dc1_inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "auth/token/renew-self" {
  capabilities = [ "update" ]
}

path "auth/token/lookup-self" {
  capabilities = [ "read" ]
}

Save the configuration in a file named vault-policy-connect-ca.hcl and create the policy with the Vault CLI.

$ vault policy write connect-ca vault-policy-connect-ca.hcl

Success! Uploaded policy: connect-ca

Create Vault token

Once the policy is created, create a Vault token to use for the integration.

$ vault token create -policy=connect-ca

Key                  Value
---                  -----
token                s.hqXGLH9OhQFKVb7avZtlDHzq
token_accessor       DwK2Zvj8kHKpLd8w6xiI2AWM
token_duration       768h
token_renewable      true
token_policies       ["connect-ca" "default"]
identity_policies    []
policies             ["connect-ca" "default"]

Setup your environment

If you use a remote deployment, you can configure a local Consul binary to interact with the datacenter. Set the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a server.

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

The same applies to the ACL token to access Consul data. You can export the token with the CONSUL_HTTP_TOKEN variable.

$ export CONSUL_HTTP_TOKEN=<consul_token>

Configure Consul service mesh CA

Once Vault is configured and the necessary policies and tokens are created, configure Consul to use Vault PKI secrets engine as Consul service mesh CA.

The suggested approach for using Vault as Consul service mesh CA is to configure it directly when deploying a new Consul datacenter.

Configure Consul servers

Configure the service mesh CA using the connect stanza in the Consul configuration file.

## Service mesh CA configuration
connect {
  enabled = true
  ca_provider = "vault"
    ca_config {
        address = "<Your-Vault-address>"
        token = "<Your-Vault-token>"
        root_pki_path = "connect_root"
        intermediate_pki_path = "connect_dc1_inter"
        leaf_cert_ttl = "72h"
        rotation_period = "2160h"
        intermediate_cert_ttl = "8760h"
        private_key_type = "rsa"
        private_key_bits = 2048
    }
}

Note

The leaf_cert_ttl defines the TTL for the certificates issued by the CA for Consul service mesh. The default value is 72h but using the parameter you can tune the lease duration to certificates. The shortest TTL value is 1h.

Start Consul servers

After the configuration you can start the Consul servers making sure to include the connect configuration.

If your Consul datacenter is already deployed, you most likely have it configured to use Consul embedded CA capabilities to generate certificates.

In this case, you can change the service mesh CA directly at runtime without the need for restarts or downtime.

Verify existing configuration

Verify your configuration using either the Consul CLI or the REST API.

$ consul connect ca get-config

{
    "Provider": "consul",
    "Config": {
        "IntermediateCertTTL": "8760h",
        "LeafCertTTL": "72h",
        "RotationPeriod": "2160h"
    },
    "State": null,
    "CreateIndex": 7,
    "ModifyIndex": 7
}

$ curl -s \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/connect/ca/configuration

{
  "Provider": "consul",
  "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RotationPeriod": "2160h"
  },
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 7,
  "ModifyIndex": 7
}

Change Consul service mesh CA configuration

Once the datacenter is operational, subsequent changes to the ca_provider configuration is ignored even if consul reload is ran or if the servers are restarted.

To update any settings under this key, you must use Consul's update CA configuration API or the consul connect ca set-config command.

First, create a file containing the new configuration for your datacenter CA.

{
  "Provider": "vault",
  "Config": {
      "Address": "<Your-Vault-address>",
      "Token": "<Your-Vault-token>",
      "RootPKIPath": "connect_root",
      "IntermediatePKIPath": "connect_dc1_inter",
      "LeafCertTTL": "72h",
      "RotationPeriod": "2160h",
      "IntermediateCertTTL": "8760h",
      "PrivateKeyType": "rsa",
      "PrivateKeyBits": 2048
  },
  "ForceWithoutCrossSigning": false
}

{
    "Provider": "vault",
    "Config": {
        "Address": "<Your-Vault-address>",
        "Token": "<Your-Vault-token>",
        "RootPKIPath": "connect_root",
        "IntermediatePKIPath": "connect_dc1_inter",
        "LeafCertTTL": "72h",
        "RotationPeriod": "2160h",
        "IntermediateCertTTL": "8760h",
        "PrivateKeyType": "ec",
        "PrivateKeyBits": 256
    },
    "ForceWithoutCrossSigning": false
}

Next, apply the configuration to your Consul datacenter.

$ consul connect ca set-config -config-file config-connect-ca-provider-vault.json

Configuration updated!

$ curl -s \
  --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
  --request PUT \
  --data @config-connect-ca-provider-vault.json \
${CONSUL_HTTP_ADDR}/v1/connect/ca/configuration

The command produces no output.

The configuration change will trigger a root certificate rotation. An intermediate CA certificate is requested during rotation from the new root, which is then cross-signed by the old root and distributed alongside the newly generated leaf certificates.

The cross-signing provides a chain of trust back to the old root certificate. This ensures new certificates are accepted also by proxies that have not completed the update to the new CA and avoids disruptions.

After the configuration is changed, you should receive the following line in your server logs.

[INFO]  agent.server.connect: CA rotated to new root under provider: provider=vault

Verify Consul service mesh CA configuration

Verify the CA settings by querying Consul for the CA configuration using either the Consul CLI or the REST API.

$ consul connect ca get-config

{
    "Provider": "vault",
    "Config": {
        "Address": "http://172.19.0.2:8200",
        "IntermediateCertTTL": "8760h",
        "IntermediatePKIPath": "connect_dc1_inter",
        "LeafCertTTL": "72h",
        "RootPKIPath": "connect_root",
        "RotationPeriod": "2160h",
        "Token": "password"
    },
    "State": null,
    "CreateIndex": 7,
    "ModifyIndex": 7
}

Configure Consul clients

Once you configured Consul service mesh for the servers, you can configure the clients.

The configuration for the clients does not require to specify the provider but only to enable the service mesh feature.

connect {
  enabled = true
}

Note

The client configuration is not tied to the CA provider in use by your Consul datacenter. This means that if you are changing the CA for an existing Consul datacenter, you will not need to apply any change to the already running clients.

Verify certificates

Once Vault is configured to act as CA for Consul service mesh you can verify the certificates being used in your service mesh.

Check root and intermediate certificates

Use the /agent/connect/ca/roots endpoint to retrieve the list of currently trusted root certificates.

$ curl -s \
  --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/roots

{
  "ActiveRootID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
  "TrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul",
  "Roots": [
    {
      "ID": "d6:65:53:7c:16:df:0c:61:33:da:22:b2:55:01:34:3d:92:2d:cf:21",
      "Name": "Consul CA Root Cert",
      "SerialNumber": 9,
      "SigningKeyID": "bb:70:0e:1a:d3:33:3e:f0:d0:0b:36:22:dc:32:2f:1b:0d:41:24:47:fc:4d:d6:43:91:da:aa:19:ce:b9:b9:f3",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:39:00Z",
      "NotAfter": "2031-07-22T13:39:00Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----\n",
      "IntermediateCerts": null,
      "Active": false,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 11,
      "ModifyIndex": 54
    },
    {
      "ID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
      "Name": "Vault CA Root Cert",
      "SerialNumber": 17018489092300240000,
      "SigningKeyID": "60:db:cb:99:95:0d:39:b7:66:66:12:fa:6b:3b:24:39:9b:4f:ae:ad",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:38:59Z",
      "NotAfter": "2021-08-23T13:39:29Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
      "IntermediateCerts": [
        "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----"
      ],
      "Active": true,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 54,
      "ModifyIndex": 54
    }
  ]
}

When a datacenter is initialized, this will only list one trusted root. As in the example above, multiple roots may appear as part of certificate rotation or if the CA got migrated from Consul to Vault.

Check leaf certificates for services

Use the /agent/connect/ca/leaf/<service> endpoint to retrieve the leaf certificates for the Consul service mesh services.

For example, to retrieve the certificate for the service web you can use the following.

$ curl -s \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

Certificate rotation and API caching

The /agent/connect/ca/leaf/<service> endpoint has a caching mechanism that might return stale results for service certificates. You can verify the result's age by inspecting the header data using the -verbose option with the curl command.

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

...
< HTTP/2 200
< age: 398999
< content-type: application/json
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-default-acl-policy: deny
< x-consul-index: 1075
...

{
  "SerialNumber": "0e",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://7d131ccc-0605-10a2-cca2-c3b4eb89500f.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-22T13:39:07Z",
  "ValidBefore": "2021-07-22T14:39:37Z",
  "CreateIndex": 58,
  "ModifyIndex": 58
}

You can verify that the result is served by the cache using the x-cache header that, in this example shows a value of HIT.

In case you notice that the age header is bigger than the value defined for leaf_cert_ttl you can invalidate the cache by pointing at a different Consul node.

Change the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a different server.

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

Re-issue the curl command to query the API endpoint. Leave the -verbose parameter set so to verify the cache headers.

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

...
< HTTP/2 200
< content-type: application/json
< vary: Accept-Encoding
< x-cache: MISS
< x-consul-default-acl-policy: deny
< x-consul-index: 486
...

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

This time you can verify that the x-cache value is MISS, meaning that the result is not coming from the cache, and you can verify from the output that the certificates returned are different from the ones returned by the cached request.

Certificate rotation in logs

You can verify the rotation is successfully happening also from Consul logs.

Server agents

When intermediate certificates for the service mesh get rotated, you will get that signaled on the leader's log.

[INFO]  agent.server.connect: generated new intermediate certificate for primary datacenter
[INFO]  agent.server.connect: updated root certificates from primary datacenter

Client agents

Client agents will get log lines for the rotation of intermediate certificates and the rotation of the service leaf certificate. The log level for those events are DEBUG, so you have to set up that verbosity level to troubleshoot the certificate rotation.

Intermediate certificate rotation:

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=roots
[DEBUG] agent.auto_config: roots watch fired - updating CA certificates

Leaf certificate rotation:

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=leaf
[DEBUG] agent.auto_config: leaf certificate watch fired - updating TLS certificate

Next Steps

In this tutorial, you configured Consul service mesh's certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul service mesh backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.

Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.

Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.

Rotate gossip encryption keys with Vault

mTLS certificates with Vault

This tutorial also appears in:

11 tutorials

Operate Consul
Ensure Consul resilience with disaster recovery best practices.
- Consul