Vault as the Controller and Connect Inject Webhook Certificate Provider on Kubernetes

This topic describes how to configure the Consul Helm chart to use TLS certificates issued by Vault in the Consul controller and connect inject webhooks.

Overview

In a Consul Helm chart configuration that does not use Vault, webhook-cert-manager ensures that a valid certificate is updated to the mutatingwebhookconfiguration of either the controller or connect inject to ensure that Kubernetes can communicate with each of these services.

When Vault is configured as the controller and connect inject Webhook Certificate Provider on Kubernetes:

webhook-cert-manager is no longer deployed to the cluster.
Controller and connect inject each get their webhook certificates from its own Vault PKI mount via the injected Vault Agent.
Controller and connect inject each need to be configured with its own Vault Role that has necessary permissions to receive certificates from its respective PKI mount.
Controller and connect inject each locally update its own mutatingwebhookconfiguration so that Kubernetes can relay events.
Vault manages certificate rotation and rotates certificates to each webhook.

To use Vault as the controller and connect inject Webhook Certificate Provider, we will need to modify the steps outlined in the Data Integration section:

These following steps will be repeated for each datacenter:

Create a Vault policy that authorizes the desired level of access to the secret.
(Added) Create Vault PKI roles for controller and connect inject that each establish the domains that each is allowed to issue certificates for.
Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
Configure the Vault Kubernetes auth roles in the Consul on Kubernetes helm chart.

Prerequisites

Complete the following prerequisites prior to implementing the integration described in this topic:

Verify that you have completed the steps described in Systems Integration section of Vault as a Secrets Backend.
You should be familiar with the Data Integration Overview section of Vault as a Secrets Backend.
Configure Vault as the Server TLS Certificate Provider on Kubernetes
Configure Vault as the Service Mesh Certificate Provider on Kubernetes

Bootstrapping the PKI Engines

Issue the following commands to enable and configure the PKI Secrets Engine to serve TLS certificates for the controller and connect inject webhooks:

Mount the PKI Secrets Engine for each:

$ vault secrets enable -path=controller pki

$ vault secrets enable -path=connect-inject pki

Tune the engine mounts to enable longer TTL:

$ vault secrets tune -max-lease-ttl=87600h controller

$ vault secrets tune -max-lease-ttl=87600h connect-inject

Generate the root CA for each:

$ vault write -field=certificate controller/root/generate/internal \
        common_name="<helm release name>-controller-webhook" \
        ttl=87600h

$ vault write -field=certificate connect-inject/root/generate/internal \
        common_name="<helm release name>-connect-injector" \
        ttl=87600h

Create Vault Policies

Create a policy that allows ["create", "update"] access to the certificate issuing URL so Consul controller and connect inject can fetch a new certificate/key pair and provide it to the Kubernetes mutatingwebhookconfiguration.
The path to the secret referenced in the path resource is the same value that you will configure in the global.secretsBackend.vault.controller.tlsCert.secretName and global.secretsBackend.vault.connectInject.tlsCert.secretName Helm configuration (refer to Update Consul on Kubernetes Helm chart).
```
$ vault policy write controller-tls-policy - <<EOF
path controller/issue/controller-role {
  capabilities = ["create", "update"]
}
EOF
```
```
$ vault policy write connect-inject-policy - <<EOF
path connect-inject/issue/connect-inject-role {
  capabilities = ["create", "update"]
}
EOF
```
Create a policy that allows ["read"] access to the CA URL, this is required for the Consul components to communicate with the Consul servers in order to fetch their auto-encryption certificates.
The path to the secret referenced in the path resource is the same values that you will configure in the global.secretsBackend.vault.controller.caCert.secretName and global.secretsBackend.vault.connectInject.caCert.secretName Helm configuration (refer to Update Consul on Kubernetes Helm chart).
```
$ vault policy write controller-ca-policy - <<EOF
path controller/cert/ca {
  capabilities = ["read"]
}
EOF
```
```
$ vault policy write connect-inject-ca-policy - <<EOF
path connect-inject/cert/ca {
  capabilities = ["read"]
}
EOF
```

Configure allowed domains for PKI certificates.

$ vault write controller/roles/controller-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

$ vault write connect-inject/roles/connect-inject-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

To generate the <Allowed-domains-string> for each use the following script as a template:

#!/bin/sh

# NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
export NAME=consulk8s
# NAMESPACE is where the Consul on Kubernetes is installed
export NAMESPACE=consul
# DATACENTER is the value of `global.datacenter` from your Helm values config file
export DATACENTER=dc1

echo allowed_domains_controller=\"${NAME}-controller-webhook,${NAME}-controller-webhook.${NAMESPACE},${NAME}-controller-webhook.${NAMESPACE}.svc,${NAME}-controller-webhook.${NAMESPACE}.svc.cluster.local\""

echo allowed_domains_connect_inject=\"${NAME}-connect-injector,${NAME}-connect-injector.${NAMESPACE},${NAME}-connect-injector.${NAMESPACE}.svc,${NAME}-connect-injector.${NAMESPACE}.svc.cluster.local\""

Finally, Kubernetes auth roles need to be created for controller and connect inject webhooks.

The path to the secret referenced in the path resource is the same values that you will configure in the global.secretsBackend.vault.controllerRole and global.secretsBackend.vault.connectInjectRole Helm configuration (refer to Update Consul on Kubernetes Helm chart).

Role for Consul controller webhooks:

$ vault write auth/kubernetes/role/controller-role \
    bound_service_account_names=<Consul controller service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=controller-ca-policy \
    ttl=1h

To find out the service account name of the Consul controller, you can run:

$ helm template --release-name ${RELEASE_NAME} --show-only templates/controller-serviceaccount.yaml hashicorp/consul -f values.yaml

Role for Consul connect inject webhooks:

$ vault write auth/kubernetes/role/connect-inject-role \
    bound_service_account_names=<Consul connect inject service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=connect-inject-ca-policy \
    ttl=1h

To find out the service account name of the Consul connect inject, use the command below.

$ helm template --release-name ${RELEASE_NAME} --show-only templates/connect-inject-serviceaccount.yaml hashicorp/consul -f values.yaml

Update Consul on Kubernetes Helm chart

Now that we've configured Vault, you can configure the Consul Helm chart to use the Server TLS certificates from Vault:

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: "consul-server"
      consulClientRole: "consul-client"
      consulCARole: "consul-ca"
      controllerRole: "controller-role"
      connectInjectRole: "connect-inject-role"
      connectInject:
        caCert:
          secretName: "connect-inject/cert/ca"
        tlsCert:
          secretName: "connect-inject/issue/connect-inject-role"
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "pki/cert/ca"
server:
  serverCert:
    secretName: "pki/issue/consul-server"
  extraVolumes:
    - type: "secret"
      name: <vaultCASecret>
      load: "false"
connectInject:
  enabled: true

The vaultCASecret is the Kubernetes secret that stores the CA Certificate that is used for Vault communication. To provide a CA, you first need to create a Kubernetes secret containing the CA. For example, you may create a secret with the Vault CA like so:

$ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/