Configure Service Mesh Certificate Authority for Consul on Kubernetes

If connect is enabled, the built-in Consul CA provider is automatically enabled for the service mesh certificate authority (CA). You can use different CA providers with Consul service mesh. Refer to Service Mesh Certificate Management for supported providers.

Overview

You should only complete the following instructions during the initial cluster bootstrapping procedure with Consul K8s CLI 0.38.0 or later. To update the Consul service mesh CA provider on an existing cluster or to update any provider properties, such as tokens, refer to Update CA Configuration Endpoint.

To configure an external CA provider using the Consul Helm chart, complete the following steps:

Create a configuration file containing your provider information.
Create a Kubernetes secret containing the configuration file.
Reference the Kubernetes secret in the server.extraVolumes value in the Helm chart.

To configure the Vault service mesh provider, refer to Vault as the Service Mesh Certificate Provider on Kubernetes.

Configuring Vault as a Service Mesh CA (Consul K8s 0.37.0 and earlier)

If you use Vault 1.11.0+ as Consul's service mesh CA, versions of Consul released before Dec 13, 2022 will develop an issue with Consul control plane or service mesh communication (GH-15525). Use or upgrade to a Consul version that includes the fix to avoid this problem.

The following instructions are only valid for Consul K8s CLI 0.37.0 and prior. It describes how to configure Vault as the service mesh CA. You can configure other providers during initial bootstrap of the cluster by providing the appropriate ca_config and ca_provider values for your provider.

Auto-renewal: If using Vault as your service mesh CA, we strongly recommend Consul 1.8.5 or later, which includes support for token auto-renewal. If the Vault token is renewable, then Consul automatically renews the token periodically. Otherwise, you must manually rotate the Vault token before it expires.

Primary Datacenter

To configure Vault as a CA provider for Consul service mesh, first, create a provider configuration JSON file. Please refer to Vault as a service mesh CA for the configuration options. You will need to provide a Vault token to the token property. Please refer to these docs for the permissions that the token needs to have. This token should be renewable.

To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/ca

And then reference it like this in the provider configuration:

{
  "connect": [
    {
      "ca_config": [
        {
          "address": "https://vault:8200",
          "intermediate_pki_path": "dc1/connect-intermediate",
          "root_pki_path": "connect-root",
          "token": "s.VgQvaXl8xGFO1RUxAPbPbsfN",
          "ca_file": "/consul/userconfig/vault-ca/vault.ca"
        }
      ],
      "ca_provider": "vault"
    }
  ]
}

This example configuration file is pointing to a Vault instance running in the same Kubernetes cluster, which has been deployed with TLS enabled. Note that the ca_file is pointing to the file location based on the Kubernetes secret for the Vault CA that we have created before. We will provide that secret later in the Helm values for our Consul cluster.

NOTE: If you have used Kubernetes CA to sign Vault's certificate, such as shown in Standalone Server with TLS, you don't need to create a Kubernetes secret with Vault's CA and can reference the CA directly by setting ca_file to /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

Next, create a Kubernetes secret with this configuration file.

$ kubectl create secret generic vault-config --from-file=config=vault-config.json

We will provide this secret and the Vault CA secret, to the Consul server via the server.extraVolumes Helm value.

global:
  name: consul
server:
  extraVolumes:
    - type: secret
      name: vault-config
      load: true
      items:
        - key: config
          path: vault-config.json
    - type: secret
      name: vault-ca
      load: false
connectInject:
  enabled: true

Finally, install the Helm chart using the above config file:

$ helm install consul --values values.yaml hashicorp/consul

Verify that the CA provider is set correctly:

$ kubectl exec consul-server-0 -- curl --silent http://localhost:8500/v1/connect/ca/configuration\?pretty
{
  "Provider": "vault",
  "Config": {
    "Address": "https://vault:8200",
    "CAFile": "/consul/userconfig/vault-server-tls/vault.ca",
    "IntermediateCertTTL": "8760h",
    "IntermediatePKIPath": "connect-intermediate",
    "LeafCertTTL": "72h",
    "RootPKIPath": "connect-root",
    "Token": "s.VgQvaXl8xGFO1RUxAPbPbsfN"
  },
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 5,
  "ModifyIndex": 5
}

Secondary Datacenters

To configure Vault as the service mesh CA in secondary datacenters, you need to make sure that the Root CA is the same, but the intermediate is different for each datacenter. In the connect configuration for a secondary datacenter, you can specify a intermediate_pki_path that is, for example, prefixed with the datacenter for which this configuration is intended. You will similarly need to create a Vault token and a Kubernetes secret with Vault's CA in each secondary Kubernetes cluster.

{
  "connect": [
    {
      "ca_config": [
        {
          "address": "https://vault:8200",
          "intermediate_pki_path": "dc2/connect-intermediate",
          "root_pki_path": "connect-root",
          "token": "s.VgQvaXl8xGFO1RUxAPbPbsfN",
          "ca_file": "/consul/userconfig/vault-ca/vault.ca"
        }
      ],
      "ca_provider": "vault"
    }
  ]
}

Note that all secondary datacenters need to have access to the same Vault instance as the primary.

Manually Rotating Vault Tokens

If running Consul < 1.8.5 or using a Vault token that is not renewable then you will need to manually renew or rotate the Vault token before it expires.

Rotating Vault Token

The ca_config and ca_provider options defined in the Consul agent configuration are only used when initially bootstrapping the cluster. Once the cluster is running, subsequent changes to the ca_provider config are ignored–even if consul reload is run or the servers are restarted.

To update any settings under these keys, you must use Consul's Update CA Configuration API or the consul connect ca set-config command.

Renewing Vault Token

To renew the Vault token, use the vault token renew CLI command or API.