HashiConf 2024 Now streaming live from Boston! Attend for free
Vault
Vault

Install Vault to Kubernetes with TLS enabled

  • 13min
  • |
  • Vault

This tutorial walks through setting up end-to-end TLS on a HA Vault cluster in Kubernetes. You will create a private key and a wildcard certificate using the Kubernetes CA. You will store the certificate and the key in the Kubernetes secrets store. Finally you will configure the Helm chart to use the Kubernetes secret.

Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, minikube, the Vault Helm chart, and the additional configuration to bring it all together.

First, follow the directions to install minikube, including VirtualBox or similar.

Next, install kubectl CLI and Helm CLI.

NOTE: This tutorial was last tested in November 2022 on a macOS 12.6.1 using this configuration.

Docker version.

$ docker version
Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
  ## ...

Display minikube version.

$ minikube version
minikube version: v1.27.1
commit: fe869b5d4da11ba318eb84a3ac00f336411de7ba

Helm version.

$ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Start minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters running inside Virtual Machines (VM) on your local system.

  1. Start a Kubernetes cluster.

    $ minikube start
😄  minikube v1.27.1 on Darwin 12.6.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.25.2 preload ...
 > preloaded-images-k8s-v18-v1...:  320.84 MiB / 320.84 MiB  100.00% 23.57 M
 > gcr.io/k8s-minikube/kicbase:  348.47 MiB / 348.47 MiB  100.00% 16.91 MiB 
 > gcr.io/k8s-minikube/kicbase:  0 B [________________________] ?% ? p/s 21s
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.25.2 on Docker 20.10.18 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

    The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

  2. Verify the status of the minikube cluster.

    $ minikube status
minikube
type: Control Plane 
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install the Vault Helm chart

Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.

  1. Add the HashiCorp Helm repository.

    $ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

  2. Update all the repositories to ensure helm is aware of the latest versions.

    $ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

  3. To verify, search repositories for vault in charts.

    $ helm search repo hashicorp/vault
NAME            CHART VERSION   APP VERSION     DESCRIPTION
hashicorp/vault 0.22.1          1.12.0          Official HashiCorp Vault Chart

Create the certificate

  1. create a working directory

    $ mkdir /tmp/vault

  2. Export the working directory location and the naming variables. 

    $ export VAULT_K8S_NAMESPACE="vault" \
export VAULT_HELM_RELEASE_NAME="vault" \
export VAULT_SERVICE_NAME="vault-internal" \
export K8S_CLUSTER_NAME="cluster.local" \
export WORKDIR=/tmp/vault

  3. Generate the private key

    $ openssl genrsa -out ${WORKDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus
....................+++
....+++

Create the Certificate Signing Request (CSR).

  1. Create the CSR configuration file

    $ cat > ${WORKDIR}/vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_SERVICE_NAME}
DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
DNS.3 = *.${VAULT_K8S_NAMESPACE}
IP.1 = 127.0.0.1
EOF

  2. Generate the CSR

    $ openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf

Issue the Certificate.

  1. Create the csr yaml file to send it to Kubernetes.

    $ cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
   name: vault.svc
spec:
   signerName: kubernetes.io/kubelet-serving
   expirationSeconds: 8640000
   request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')
   usages:
   - digital signature
   - key encipherment
   - server auth
EOF

  2. Send the CSR to Kubernetes

    $ kubectl create -f ${WORKDIR}/csr.yaml
certificatesigningrequest.certificates.k8s.io/vault.svc created

  3. Approve the CSR in Kubernetes.

    $ kubectl certificate approve vault.svc
certificatesigningrequest.certificates.k8s.io/vault.svc approved

  4. Confirm the certificate was issued

    $ kubectl get csr vault.svc
NAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITION
vault.svc   16s   kubernetes.io/kubelet-serving   minikube-user   100d                Approved,Issued

Store the certificates and Key in the Kubernetes secrets store

  1. Retrieve the certificate

    $ kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt

  2. Retrieve Kubernetes CA certificate

    $ kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > ${WORKDIR}/vault.ca

  3. Create the Kubernetes namespace

    $ kubectl create namespace $VAULT_K8S_NAMESPACE
namespace/vault created

  4. Create the TLS secret

    $ kubectl create secret generic vault-ha-tls \
   -n $VAULT_K8S_NAMESPACE \
   --from-file=vault.key=${WORKDIR}/vault.key \
   --from-file=vault.crt=${WORKDIR}/vault.crt \
   --from-file=vault.ca=${WORKDIR}/vault.ca

    Output:

    # secret/vault-ha-tls created

Deploy the vault cluster via Helm with overrides

  1. Create the overrides.yaml file.

    $ cat > ${WORKDIR}/overrides.yaml <<EOF
global:
   enabled: true
   tlsDisable: false
injector:
   enabled: true
server:
   extraEnvironmentVars:
      VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
      VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
      VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true
   standalone:
      enabled: false
   affinity: ""
   ha:
      enabled: true
      replicas: 3
      raft:
         enabled: true
         setNodeId: true
         config: |
            cluster_name = "vault-integrated-storage"
            ui = true
            listener "tcp" {
               tls_disable = 0
               address = "[::]:8200"
               cluster_address = "[::]:8201"
               tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
               tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
               tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
            }
            storage "raft" {
               path = "/vault/data"
            }
            disable_mlock = true
            service_registration "kubernetes" {}
EOF

    Recommendation

    If you are using Prometheus for monitoring and alerting, we recommend to set the cluster_name in the HCL configuration. With the Vault Helm chart, this is accomplished with the config parameter.

  2. Deploy the Cluster

    $ helm install -n $VAULT_K8S_NAMESPACE $VAULT_HELM_RELEASE_NAME hashicorp/vault -f ${WORKDIR}/overrides.yaml

    Example output: 

    NAME: vault
LAST DEPLOYED: Thu Nov  3 19:47:36 2022
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

   $ helm status vault
   $ helm get manifest vault

  3. Display the pods in the namespace that you created for vault

    $ kubectl -n $VAULT_K8S_NAMESPACE get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          22s
vault-1                                 0/1     Running   0          22s
vault-2                                 0/1     Running   0          22s
vault-agent-injector-6679b665fd-92n5q   1/1     Running   0          23s

  4. Initialize vault-0 with one key share and one key threshold.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > ${WORKDIR}/cluster-keys.json

    The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a file named cluster-keys.json.

  5. Display the unseal key found in cluster-keys.json.

    $ jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.json
ezMnsvK9j6aArz1sP0kQNBnTW5PH5u15iFtwPp4Jdzg=

    Insecure operation

    Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.

  6. Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.

    $ VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" ${WORKDIR}/cluster-keys.json)

    After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

  7. Unseal Vault running on the vault-0 pod.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

    Insecure operation

    Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.

    The operator unseal command reports that Vault is initialized and unsealed.

    Example output: 

    Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.12.0
Build Date              2022-10-10T18:14:33Z
Storage Type            raft
Cluster Name            vault-cluster-5b1b8251
Cluster ID              2e2f214b-c4e7-39e7-52b5-39bffba1aa12
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-11-04T13:08:48.505370004Z
Raft Committed Index    36
Raft Applied Index      36

    The Vault server is initialized and unsealed.

Join vault-1 and vault2 pods to the Raft cluster

  1. Start an interactive shell session on the vault-1 pod. 

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-1 -- /bin/sh
/ $

    Your system prompt is replaced with a new prompt / $.

  2. Join the vault-1 pod to the Raft cluster. 

    $ vault operator raft join -address=https://vault-1.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

    Example output: 

    Key       Value
---       -----
Joined    true

  3. Exit the vault-1 pod.

    $ exit

  4. Unseal vault-1.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.0
Build Date         2022-10-10T18:14:33Z
Storage Type       raft
HA Enabled         true

  5. Start an interactive shell session on the vault-2 pod. 

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-2 -- /bin/sh
/ $

    Your system prompt is replaced with a new prompt / $.

  6. Join the vault-2 pod to the Raft cluster. 

    $ vault operator raft join -address=https://vault-2.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

    Example output: 

    Key       Value
---       -----
Joined    true

  7. Exit the vault-2 pod.

    $ exit

  8. Unseal vault-2.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.12.0
Build Date         2022-10-10T18:14:33Z
Storage Type       raft
HA Enabled         true

    Login to vault and confirm everything is working

  9. Export the cluster root token

    $ export CLUSTER_ROOT_TOKEN=$(cat ${WORKDIR}/cluster-keys.json | jq -r ".root_token")

  10. Login to vault-0 with the root token

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault login $CLUSTER_ROOT_TOKEN

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.lF3yPwVSCl6AwMI7XdrsirdH
token_accessor       JaQ0MUL3R1zjbTSkUMCKARmk
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

  11. List the raft peers.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator raft list-peers

Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
vault-2    vault-2.vault-internal:8201    follower    true

  12. Print the HA status

    $ kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault status

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.12.0
Build Date              2022-10-10T18:14:33Z
Storage Type            raft
Cluster Name            vault-cluster-0a1e62ef
Cluster ID              95972f8a-c38e-8200-9a67-0a910b34b691
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-11-04T14:21:05.668816678Z
Raft Committed Index    42
Raft Applied Index      42

    You now have a working 3 node cluster with TLS enabled at the pod level. Next you will create a secret and retrieve it via and API call to confirm TLS is working as expected.

Create a secret

  1. Start an interactive shell session on the vault-0 pod.

    $ kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-0 -- /bin/sh
/ $

    Note

    Your system prompt is replaced with a new prompt / $.

  2. Enable the kv-v2 secrets engine

    $ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

  3. Create a secret at the path secret/tls/apitest with a username and a password

    $ vault kv put secret/tls/apitest username="apiuser" password="supersecret"

===== Secret Path =====
secret/data/tls/apitest

======= Metadata =======
Key                Value
---                -----
created_time       2022-11-07T17:00:23.335480752Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

  4. Verify that the secret is defined at the path secret/tls/apitest

    $ vault kv get secret/tls/apitest

===== Secret Path =====
secret/data/tls/apitest

======= Metadata =======
Key                Value
---                -----
created_time       2022-11-07T17:00:23.335480752Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    supersecret
username    apiuser

  5. Exit the vault-0 pod.

    $ exit

Expose the vault service and retrieve the secret via the API

The Helm chart defined a Kubernetes service named vault that forwards requests to its endpoints (i.e. The pods named vault-0, vault-1, and vault-2).

  1. Confirm the Vault service configuration

    $ kubectl -n $VAULT_K8S_NAMESPACE get service vault

NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
vault   ClusterIP   10.110.24.17   <none>        8200/TCP,8201/TCP   68m

  2. In another terminal, port forward the vault service.

    $ kubectl -n vault port-forward service/vault 8200:8200

Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

  3. In the original terminal, perform a HTTPS curl request to retrieve the secret you created in the previous section.

    $ curl --cacert $WORKDIR/vault.ca \
   --header "X-Vault-Token: $CLUSTER_ROOT_TOKEN" \
   https://127.0.0.1:8200/v1/secret/data/tls/apitest | jq .data.data

    Example output:

    {
"password": "supersecret",
"username": "apiuser"
}

    The secret you created earlier is displayed back to us.

Clean up

  1. Stop the running local Kubernetes cluster.

    $ minikube stop

✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

    This deactivates minikube, and all pods still exist at this point.

  2. Delete the local Kubernetes cluster.

    Be aware that minikube delete removes the minikube deployment including all pods. Be sure you want everything removed before continuing. 

    $ minikube delete

🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/username/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

  3. Remove the files and the working directory you created.

    $ rm -r $WORKDIR

Next steps

You launched Vault in high-availability via a Helm chart with TLS enabled. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

Then you created and API call and requested a secret directly from Vault. Explore how pods can retrieve secrets through the Vault Injector service via annotations or secrets mounted on ephemeral volumes.

Previous

Vault installation to minikube with Consul

 Next

Vault installation to Amazon EKS via Helm