HashiConf 2024 Now streaming live from Boston! Attend for free
Vault
Vault

Dynamic credentials for Kubernetes workloads with Vault and VSO

  • 43min
  • |
  • Enterprise
  • Vault
  • Interactive

Operating system

The Vault Secrets Operator (VSO) is a Kubernetes Operator that allows teams to centrally manage secrets in Vault and synchronize them with native Kubernetes Secrets.

Scenario

HashiCups, the world's finest coffee roaster, has recently acquired International Brewing Machines (IBM) who makes smart coffee roasters. The teams at HashiCups and IBM need to make the app compliant with HashiCups security standards.

HashiCups recently implemented HashiCorp Vault as part of their security lifecycle management processes. HashiCups requires applications to use short-lived, time bound credentials to limit potential exposure of credentials.

The application is a Kubernetes native application which consumes Kubernetes native secrets. These secrets are manually provisioned, and do not expire.

Alice from the HashiCups architect team meets with Watson from the IBM development team to figure out how they can support the app.

Challenge

The app runs on Kubernetes and retrieves secrets directly from manually provisioned Kubernetes Secrets. The Kubernetes Secret has a username and password for the app to connect to a PostgreSQL database.

The team at HashiCups has the following requirements:

  • Replace static, long lived credentials with short-lived, time bound credentials credentials.
  • Allow teams to securely store API keys, passwords, certificates, etc.
  • Onboard the app quickly, as the teams do not have time to refactor the app.

Solution

To meet the requirement of on-boarding the app quickly, Alice proposes using the Vault Secrets Operator. Alice and Watson decide to configure a static secret in the Vault K/V v2 secrets engine as a proof-of-concept for deploying the VSO. After a successful test with static secrets, they will configure Vault to generate dynamic credentials for PostgreSQL and configure VSO to work with dynamic secrets.

What is the Vault Secrets Operator

Vault provides a complete solution for modern secrets management. It consolidates identity brokering, secrets management, dynamic secrets, rotation, and security policy compliance in one platform. By coupling Vault’s secret management feature set with a Kubernetes Secrets, developers do not have to refactor their applications and can continue to reference Kubernetes Secrets while security and platform teams can manage the lifecycle of secrets through Vault.

How the Vault Secrets Operator works

As a controller, VSO reconciles the current state of secrets defined in the cluster to the desired state specified by custom resources using standard Kubernetes declarative patterns.

You deploy The Vault Secrets Operator using Helm or Kustomize.

The Vault Secrets Operator requires the following Kubernetes permissions:

ObjectPermissionReason
Secretcreate, read, update, delete, watchSync operations, Vault auth
ServiceAccountread, token creationVault auth
Deploymentread, update, watchPost secret rotation actions

Prerequisites

This lab was tested on macOS 14.6.1 and Windows 10 with PowerShell Core 7.4.5.

Set up the lab

To complete this lab, you will deploy the following:

  • Kubernetes cluster using minikube.

  • Vault cluster external to the Kubernetes cluster configured to authenticate with the Kubernetes API.

  • PostgreSQL with Vault configured to create on-demand PostgreSQL roles. Diagram showing an example Kubernetes and Vault deployment with the Vault Secrets Operator in a dedicated namespace Diagram showing an example Kubernetes and Vault deployment with the Vault Secrets Operator in a dedicated namespace

    If you are unable to install the required tools to complete this lab, you can follow this embeded lab for a similar experience.

    Launch Terminal

    This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Note

Command output may differ slightly when using Microsoft Windows.

  1. Launch PowerShell and create a working directory to store files created in the lab.

    New-Item -Path ~/HC_LAB -ItemType "directory"

  2. Change to the lab directory.

    Set-Location -Path ~/HC_LAB

    Leave PowerShell open for the remainder of the lab.

Start and configure Kubernetes

You will use minikube, a CLI tool that provisions and manages the lifecycle of single-node Kubernetes cluster, to set up a Kubernetes cluster on your system.

  1. Start a Kubernetes cluster.

    $ minikube start

😄  minikube v1.25.2 on Darwin 12.3
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

    The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

  2. Verify the status of the Kubernetes cluster.

    $ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

  3. Open a new terminal and start a proxy to expose the Kubernetes API.

    $ kubectl proxy --disable-filter=true
W0822 12:57:23.605710   10659 proxy.go:177] Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on 127.0.0.1:8001

    This command exposes the Kubernetes API to your local machine. This terminal must remain open during the lab.

  4. Open another new terminal, and create a tunnel to the proxy listening on port 8001.

    Warning

    ngrok is used to mimic connectivity of complex network configurations to expose the Kubernetes API. Using --scheme=http exposes the API without encryption to avoid TLS certificate errors.

    For production workloads, connect to a trusted network with valid TLS certificates.

    $ ngrok http --scheme=http 127.0.0.1:8001

    Example output:

    ngrok                                                                                                                                                              (Ctrl+C to quit)

Session Status                online
Account                       username (Plan: Free)
Update                        update available (version 3.x.x, Ctrl-U to update)
Version                       3.x.x
Region                        United States (us)
Latency                       32.791235ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://d12b-34-567-89-10.ngrok.io -> 127.0.0.1:8001

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

    This terminal must remain open during the lab.

    ngrok is used for simplicity in exposing the Kubernetes API from minikube which is meant to serve as a local environment. In a production Kubernetes cluster, you might use a load balancer, or managed services such as AKS or EKS expose the API by default.

  5. Copy the forwarding address and return to the terminal where you started minikube.

  6. Export an environment variable for the Kubernetes API address using the ngrok forwarding address. This address will be unique to your lab.

    Note

    Remove any trailing slashes from the URL.

    $Env:K8S_URL="<ngrok-address>"

    The Kubernetes cluster is now accessible by an external Vault cluster.

Start Vault

  1. Export an environment variable with a valid Vault Enterprise license.

    $Env:VAULT_LICENSE="your-license-key"

  2. Start Vault Enterprise in a container. Vault will operate outside the Kubernetes cluster and provides a similar experience to using HCP Vault Dedicated or HCP Vault Secrets with VSO.

    docker run --name vault-enterprise `
    --cap-add=IPC_LOCK `
    --env VAULT_LICENSE=$(Get-Content -Path env:VAULT_LICENSE) `
    --env VAULT_DEV_ROOT_TOKEN_ID=root `
    --env VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 `
    --publish 8200:8200 `
    --detach `
    --rm `
    hashicorp/vault-enterprise

    Example output:

    Unable to find image 'hashicorp/vault-enterprise:latest' locally
latest: Pulling from hashicorp/vault-enterprise
6e5015e02423: Pull complete
6be2c75e5421: Pull complete
Digest: sha256:c7e266b84e44580d1f5496c91c23e051f96c75a8774de4b413a563119768c50f
Status: Downloaded newer image for hashicorp/vault-enterprise:latest
5cb93ad2e2e9737761c0903966a0ccc8fd6a835604b30b63a2a23babc8332f59

    The Vault dev server listens on all addresses using port 8200. The server is initialized and unsealed.

    Insecure operation

    Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

  3. Verify the Vault Enterprise container is in a STATUS is Up.

    docker ps -f name=vault-enterprise

    Example output:

    CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS                    NAMES
5cb93ad2e2e9   hashicorp/vault-enterprise   "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes   0.0.0.0:8200->8200/tcp   vault-enterprise

  4. Export an environment variable for the vault CLI to address the Vault server.

    $Env:VAULT_ADDR="http://127.0.0.1:8200"

  5. Export an environment variable for the vault CLI to authenticate with the Vault server.

    $Env:VAULT_TOKEN="root"

    Note

    For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

  6. Verify Vault Enterprise is running and unsealed.

    vault status

    Example output:

    Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.17.3+ent
Build Date      2024-08-06T14:46:20Z
Storage Type    inmem
Cluster Name    vault-cluster-77c01606
Cluster ID      a9a798e7-15ee-ef5d-e62b-72a893480665
HA Enabled      false

    The Vault cluster is now ready.

Install the Vault Secrets Operator

  1. Install and update the HashiCorp Helm repository.

    helm repo add hashicorp https://helm.releases.hashicorp.com ; `
  helm repo update

    Example output:

    "hashicorp" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

  2. Install the Vault Secrets Operator in a new Kubernetes namespace.

    helm install vault-secrets-operator hashicorp/vault-secrets-operator `
  --namespace vault-secrets-operator `
  --create-namespace

    Example output:

    NAME: vault-secrets-operator
LAST DEPLOYED: Thu Jan 25 11:54:37 2024
NAMESPACE: vault-secrets-operator
STATUS: deployed
REVISION: 1

    In addition to installing the Vault Secrets Operator, the installation includes the secrets.hashicorp.com/v1beta1 API used to create Kubernetes resources.

Create a connection to Vault

VSO uses the VaultConnection as a reference to connect to the external Vault cluster.

  1. Create a Kubernetes namespace for Vault resources.

    $ kubectl create namespace vault

    A good practice is to use Kubernetes namespaces as a logical security boundary for different resources.

    The vault Kubernetes namespace stores the connection to Vault, and a service account used by Vault to authenticate with the Kubernetes API.

  2. Connect to minikube and export an environment variable with the address for the external Vault container.

    $Env:K8S_TO_VAULT=$(minikube ssh "dig +short host.docker.internal")

    In a production environment, this would be the address of your Vault cluster accessible by the Kubernetes API.

  3. Create a connection to Vault.

    $VaultConnection = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: vault
  name: vault-connection
spec:
  # address to the Vault server.
  address: 'http://$(Get-Content -Path env:K8S_TO_VAULT):8200'
---
"@
$VaultConnection | kubectl create -f -

    Example output:

    vaultconnection.secrets.hashicorp.com/vault-connection created

  4. Verify the configuration.

    $ kubectl describe --namespace vault vaultconnection.secrets.hashicorp.com/vault-connection

Name:         vault-connection
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultConnection
Metadata:
  Creation Timestamp:  2024-01-25T16:58:47Z
  Finalizers:
    vaultconnection.secrets.hashicorp.com/finalizer
  Generation:        1
  Resource Version:  3720
  UID:               60500026-9195-4e82-bb6b-9d255c1cd8dc
Spec:
  Address:          http://192.168.65.254:8200
  Skip TLS Verify:  false
Status:
  Valid:  true
Events:
  Type    Reason    Age   From             Message
  ----    ------    ----  ----             -------
  Normal  Accepted  107s  VaultConnection  VaultConnection accepted

Create a service account for Vault

  1. Create a Kubernetes service account named vault-to-k8s-sa with a service account token. Vault authenticates with the Kubernetes API using this token.

    $VaultToK8sSA = @"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-to-k8s-sa
  namespace: vault
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-to-k8s-sa-secret
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-to-k8s-sa
type: kubernetes.io/service-account-token
---
"@
$VaultToK8sSA | kubectl create -f -

    Example output:

    serviceaccount/vault-to-k8s-sa created
secret/vault-to-k8s-sa-secret created

  2. Create a Kubernetes role for the vault-to-k8s-sa service account to allow access to the Kubernetes API.

    $VaultToK8sSaTokenreviewRole = @"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-to-k8s-sa-tokenreview-role
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-to-k8s-sa
    namespace: vault
"@
$VaultToK8sSaTokenreviewRole | kubectl create -f -

    Example output:

    clusterrolebinding.rbac.authorization.k8s.io/vault-to-k8s-sa-tokenreview-role created

  3. Retrieve the ca.crt certificate from the service account secret and store it in a file.

    kubectl get secret --namespace vault vault-to-k8s-sa-secret --output json | jq -r '.data | ."ca.crt"' | base64 -d >> ca.crt

  4. Retrieve the token from the service account secret and store it in a file.

    kubectl get secret --namespace vault vault-to-k8s-sa-secret --output json | jq -r '.data | .token' | base64 -d >> token.txt

  5. Verify the files contain the required information.

    Get-Content .\token.txt, .\ca.crt

    Example output:

    eyJhbGciOiJSUzI1NiIsImtpZCI6Im1xay1Mb3ZRWndUNXowV3hLZnVsdERfTXZYU1hGZXhaNmRMS29NcDhVRU0ifQ.eyJpc3MiOiJrd
WJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia
...snip...
W4KlKrAHUxmWfkmqn-TOg6adqM6gpZ86t3hpNKLieui4k7jIn6MRfvJasuAV7POJ9ZrdKMYWnOE4Hob7H-_Jck0IVBOEpSCd4YIyLU5e
cxOIm9gYnyz7W0P2D4EnWonqPzpAn3YGi5wilRMQ6f6thN8KDz__zYn0Bqx4ioSQ

-----BEGIN CERTIFICATE-----
MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTIyMDQyNjE0MDE0NVoXDTMyMDQyNDE0MDE0NVowFTETMBEGA1UE
...snip...
tIS6p2TV6cW+Fa6VFvRYcjb4hfPovAB5gjhm36YravcjgP9rt+h0rFbQEc5oTtMa
hThaYp4czraioA==
-----END CERTIFICATE-----

    You have collected the necessary information to configure the Vault Kubernetes auth method.

Enable the Kubernetes auth method

  1. Enable the Kubernetes auth method.

    $ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

  2. Configure the Kubernetes auth method to connect to the Kubernetes API using the vault-to-k8s-sa service account token.

    vault write auth/kubernetes/config `
  token_reviewer_jwt=$(Get-Content -Path .\token.txt -Raw) `
  kubernetes_host=$(Get-Content -Path env:K8S_URL) `
  kubernetes_ca_cert=$(Get-Content -Path .\ca.crt -Raw)

    Example output:

    Success! Data written to: auth/kubernetes/config

    The Kubernetes auth method is configured and ready to proceed with the lab.

Enable KV secrets engine

Configure the KV secrets engine to verify the Vault Secrets Operator is working with a basic static secret. The KV secrets engine is useful for storing API keys, and credentials that can not be automatically generated.

  1. Enable the KV secrets engine.

    $ vault secrets enable -version=2 -path=static kv
Success! Enabled the kv secrets engine at: static/

    Alice and Watson use the KV secrets engine to validate the IBM app can get static secrets managed by Vault and synchronized to a Kubernetes secret.

  2. Create a secret at path static/exampleapp/creds with a username and password.

    vault kv put static/exampleapp/creds `
 username='jalbertson' `
 password='worstpasswordever'

    Example output:

    ======== Secret Path ========
static/data/exampleapp/creds

======= Metadata =======
Key                Value
---                -----
created_time       2024-08-22T16:49:33.977869448Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

  3. Create a Vault policy that permits read access to static/exampleapp/creds.

    $VaultPolicyExampleRead = @"
path "static/data/exampleapp/creds" {
  capabilities = ["read"]
}
"@
$VaultPolicyExampleRead | vault policy write exampleapp-kv-read -

    Example output:

    Success! Uploaded policy: exampleapp-kv-read

    The KV secrets engine and policy is ready to proceed with the lab.

Sync static secrets

Before the Vault Secrets Operator can sync secrets from Vault to Kubernetes Secrets, it needs a Vault role to authenticate with. The policy attached to the role permits access to the KV secrets engine created in the previous section.

  1. Create a Vault role and include the exampleapp-kv-read Vault policy for the Kubernetes service account vso-static-exampleapp-sa. This service account allows VSO to create secrets in the Kubernetes namespace static-exampleapp.

    vault write auth/kubernetes/role/vault-role-static-exampleapp `
bound_service_account_names=vso-static-exampleapp-sa `
bound_service_account_namespaces=static-exampleapp `
policies=exampleapp-kv-read `
ttl=1h

    Example output:

    Success! Data written to: auth/kubernetes/role/vault-role-static-exampleapp

    The bound_service_account_namespaces parameter is a list of Kubernetes namespaces allowed to call the Vault vault-role-static-exampleapp role.

    The bound_service_account_names parameter is a list of Kubernetes service accounts allowed to call the Vault vault-role-static-exampleapp role.

    You will now create a Kubernetes namespace to deploy the app. This namespace will also include a Kubernetes service account. This service account must exist in the Kubernetes namespace where you want VSO to write the secret to.

  2. Create a Kubernetes namespace to create the secret in and deploy the HashiCups/IBM app.

    kubectl create namespace static-exampleapp

    Example output:

    namespace/static-exampleapp created

  3. Create a Kubernetes service account for VSO to use to create a secret in the static-exampleapp Kubernetes namespace.

    $StaticServiceAccount = @"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vso-static-exampleapp-sa
  namespace: static-exampleapp
---
"@
$StaticServiceAccount | kubectl create -f -

    Example output:

    serviceaccount/vso-static-exampleapp-sa created

  4. Configure authentication for the Vault Secrets Operator using the vso-static-exampleapp-sa Kubernetes service account.

    $StaticServiceAccountAuth = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth-vso-static-exampleapp
  namespace: static-exampleapp
spec:
  vaultConnectionRef: vault/vault-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces:
    - static-exampleapp
  kubernetes:
    role: vault-role-static-exampleapp
    serviceAccount: vso-static-exampleapp-sa
---
"@
$StaticServiceAccountAuth | kubectl create -f -

    Example output:

    vaultauth.secrets.hashicorp.com/vault-auth-vso-static-exampleapp created

  5. Configure the Vault Secrets Operator to read from the secret KV v2 mount at the exampleapp/creds path.

    $CreateStaticSecret = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
  namespace: static-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-static-exampleapp
  mount: static
  type: kv-v2
  path:  exampleapp/creds
# version: 2
  refreshAfter: 300s
  destination:
    create: true
    name: vso-static-creds-from-vault
---
"@
$CreateStaticSecret | kubectl create -f -

    Example output:

    vaultstaticsecret.secrets.hashicorp.com/vault-static-secret created

  6. Verify the Vault secret is available as a native Kubernetes Secret.

    $ kubectl get secrets --namespace static-exampleapp

NAME                              TYPE                                  DATA   AGE
vso-static-creds-from-vault       Opaque                                3      15s

    VSO creates the secret vso-static-creds-from-vault and bases the name on the name parameter provided in the VaultStaticSecret configuration.

  7. Read the Kubernetes Secret value and decode the base64 encoded strings.

    $ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"

{
  "_raw": "{\"data\":{\"password\":\"worstpasswordever\",\"username\":\"jalbertson\"},\"metadata\":{\"created_time\":\"2024-01-25T15:48:31.871429498Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":1}}",
  "password": "worstpasswordever",
  "username": "jalbertson"
}

    Applications can now use the Kubernetes Secret by injecting it through a volume mount or an environment variable.

  8. Create a deployment for the HashiCups app that reads the Kubernetes secret and creates an environment variable.

    $DeployExampleapp = @"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hashicups-exampleapp
  namespace: static-exampleapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hashicups-exampleapp
  template:
    metadata:
      labels:
        app: hashicups-exampleapp
    spec:
      containers:
      - name: test
        image: jfrappier/dynamic-exampleapp:latest
        ports:
        - containerPort: 5000
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: vso-static-creds-from-vault
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: vso-static-creds-from-vault
              key: password
        - name: DB_HOST
          value: "your_db_host"
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: "your_db_name"
---
"@
$DeployExampleapp | kubectl create -f -

    The app references the Kubernetes Secret vso-static-creds-from-vault and injects an environment variable to the running pod.

  9. Verify the pod is in a running state. It may take a few minutes to download the image and start the pod.

    $ kubectl get pods --namespace static-exampleapp

NAME                                    READY   STATUS    RESTARTS   AGE
hashicups-exampleapp-645f8b5867-8qll7   1/1     Running   0          4m

  10. Access the applications secret endpoint to display the Kubernetes Secret created by VSO.

    kubectl exec --namespace static-exampleapp --stdin=true `
  $(kubectl get pods --namespace static-exampleapp -l app=hashicups-exampleapp -o name) `
  -- curl http://127.0.0.1:5000/secret --silent

    Example output:

    {"password":"WorstPasswordEver","username":"jalbertson"}

    The app was able to read the Kubernetes Secret created by VSO. The deployment injected the secrets as environment variables, however you can also can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.

  11. Delete the exampleapp deployment.

    $ kubectl delete deployment hashicups-exampleapp --namespace static-exampleapp

    This step is done only for lab purposes to conserve resources on your local machine.

Sync dynamic secrets

Alice and Watson have successfully tested the app with Vault and VSO. VSO creates the Kubernetes Secret from a Vault managed secret, and the app works without having to develop new features to read the Vault managed secret.

The Vault database secrets engine connects to a supported RDBMS and manages the creation and revocation of credentials. When a credentials TTL is reached, Vault revokes the credentials.

Alice and Watson will test Vault's database secrets engine, which creates dynamic, just-in-time credentials. These credentials are short lived, limiting potential exposure of the credentials.

Start and configure PostgreSQL

  1. Start PostgreSQL.

    docker run `
   --detach `
   --name learn-postgres `
   -e POSTGRES_USER=root `
   -e POSTGRES_PASSWORD=rootpassword `
   -p 5432:5432 `
   --rm `
   postgres

    Example output:

    Unable to find image 'postgres:latest' locally
latest: Pulling from library/postgres
e4fff0779e6d: Already exists
3dd23fa89c28: Pull complete
...snip...
133de8acf4aa: Pull complete
Digest: sha256:c62fdb7fd6f519ef425c54760894c74e8d0cb04fbf4f7d3d79aafd86bae24edd
Status: Downloaded newer image for postgres:latest
ebeb6f879088f172b6993dbcc67e6d46c9158d01e1bbae7c7dd17a9dd3ae45ca

  2. Connect to minikube and export an environment variable with the address for the PostgreSQL container.

    $env:K8S_TO_POSTGRESQL=$(minikube ssh "dig +short host.docker.internal")

    In a production environment, this would be the address of your PostgreSQL server accessible by Vault.

  3. Enable the database secrets engine.

    vault secrets enable database

  4. Configure the database secrets engine to use the postgresql-database-plugin, and the PostgreSQL root credentials.

    vault write database/config/postgresql `
   plugin_name=postgresql-database-plugin `
   connection_url="postgresql://{{username}}:{{password}}@$(Get-Content -Path env:K8S_TO_POSTGRESQL)/postgres?sslmode=disable" `
   allowed_roles=vault-role-dynamic-exampleapp `
   username="root" `
   password="rootpassword"

    The credentials used can be any PostgreSQL role with the ability to create and revoke PostgreSQL roles.

  5. Configure a Vault role named vault-role-dynamic-exampleapp that includes a SQL statement to create the PostgreSQL role.

    vault write database/roles/vault-role-dynamic-exampleapp `
   db_name="postgresql" `
   creation_statements="CREATE ROLE `"{{name}}`" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; `
       GRANT SELECT ON ALL TABLES IN SCHEMA public TO `"{{name}}`";" `
   default_ttl="1h" `
   max_ttl="24h"

    In production environments, a good practice is to include a valid revocation statement which are valid for the database you've configured. If you do not specify statements appropriate to creating, revoking, or rotating users, Vault inserts generic statements which can be unsuitable for your deployment.

  6. Test the Vault role and generate PostgreSQL credentials.

    vault read database/creds/vault-role-dynamic-exampleapp

    Example output:

    Key                Value
---                -----
lease_id           database/creds/vault-role-dynamic-exampleapp/JzEIFnrPGDWrxvxyvAHeLTyZ
lease_duration     1h
lease_renewable    true
password           aNUEd-agvqnclxpkAxG4
username           v-token-vault-ro-DBrelKpqZTyhuEI2lXYh-1724964233

    Vault uses the creation_statements from the vault-role-dynamic-exampleapp Vault role to create a new PostgreSQL username and password.

Create and sync dynamic secrets

With the database secrets engine configured Alice and Watson will configure VSO to create and sync the new credentials to a Kubernetes Secret.

  1. Create a policy that allows access to the database secrets engine.

    $VaultDatabasePolicy = @"
path "database/creds/vault-role-dynamic-exampleapp" {
  capabilities = [ "read" ]
}
"@
$VaultDatabasePolicy | vault policy write exampleapp-database-read -

    Example output:

    Success! Uploaded policy: exampleapp-database-read

  2. Create a Vault role for the app with the exampleapp-database-read policy attached.

    vault write auth/kubernetes/role/vault-role-dynamic-exampleapp `
bound_service_account_names=vso-dynamic-exampleapp-sa `
bound_service_account_namespaces=dynamic-exampleapp `
policies=exampleapp-database-read `
ttl=1h

    Example output:

    Success! Data written to: auth/kubernetes/role/vault-role-dynamic-exampleapp

    A good practice is to create separate Vault roles for each Kubernetes namespace.

  3. Create a Kubernetes namespace to store the dynamic secrets and run the app.

    $ kubectl create namespace dynamic-exampleapp
namespace/dynamic-exampleapp created

    A new Kubernetes namespace is used to show the steps specific to configuring VSO for dynamic secrets. Certain Kubernetes custom resources such as the VaultConnection resource can be shared across different namespaces.

  4. Create a Kubernetes service account for VSO to use to create a dynamic secret in the dynamic-exampleapp namespace.

    $DynamicExampleappNS = @"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: dynamic-exampleapp
  name: vso-dynamic-exampleapp-sa
---
"@
$DynamicExampleappNS | kubectl create -f -

    Example output:

    serviceaccount/vso-dynamic-exampleapp-sa created

  5. Configure authentication for the Vault Secrets Operator using the Kubernetes vso-static-exampleapp-sa service account and the vault-role-dynamic-exampleapp Vault role.

    $DynamicServiceAccountAuth = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth-vso-dynamic-exampleapp
  namespace: dynamic-exampleapp
spec:
  vaultConnectionRef: vault/vault-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces:
    - dynamic-exampleapp
  kubernetes:
    role: vault-role-dynamic-exampleapp
    serviceAccount: vso-dynamic-exampleapp-sa
---
"@
$DynamicServiceAccountAuth | kubectl create -f -

    Example output:

    vaultauth.secrets.hashicorp.com/vault-auth-vso-dynamic-exampleapp created

    The connection resource vaultConnectionRef is reused from earlier in the lab.

  6. Configure the Vault Secrets Operator to read from the database mount using the vault-role-dynamic-exampleapp role.

    $DynamicSecret = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: vso-db-demo-create
  namespace: dynamic-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-dynamic-exampleapp
  mount: database
  path: creds/vault-role-dynamic-exampleapp
  destination:
    create: true
    name: vso-dynamic-creds-from-vault
  rolloutRestartTargets:
  - kind: Deployment
    name: hashicups-exampleapp
---
"@
$DynamicSecret | kubectl create -f -

    Example output:

    vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo-create created

    VSO will create a Kubernetes Secret named vso-dynamic-creds-from-vault using the secrets engine mounted at the path database using the vault-role-dynamic-exampleapp Vault role.

    The rolloutRestartTargets can be configured whenever the application(s) consuming the Vault secret does not support dynamically reloading a rotated secret. In that case one, or more RolloutRestartTargets can be configured. VSO will trigger a "rollout-restart" for each target whenever the Vault secret changes between reconciliation events.

  7. Verify the Vault database credentials are available as a native Kubernetes Secret.

    kubectl get secrets --namespace dynamic-exampleapp

    Example output:

    NAME                           TYPE     DATA   AGE
vso-dynamic-creds-from-vault   Opaque   3      10s

  8. Read the Kubernetes Secret value and decode the base64 encoded strings.

    $ kubectl get secret --namespace dynamic-exampleapp vso-dynamic-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"password\":\"CWZODcdw-Di97dVWzRgd\",\"username\":\"v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758\"}",
  "password": "CWZODcdw-Di97dVWzRgd",
  "username": "v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758"
}

  9. Create a deployment for the HashiCups app that reads the Kubernetes Secret.

    $DeployDynamicExampleapp = @"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hashicups-exampleapp
  namespace: dynamic-exampleapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hashicups-exampleapp
  template:
    metadata:
      labels:
        app: hashicups-exampleapp
    spec:
      containers:
      - name: test
        image: jfrappier/dynamic-exampleapp:latest
        ports:
        - containerPort: 5000
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: vso-dynamic-creds-from-vault
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: vso-dynamic-creds-from-vault
              key: password
        - name: DB_HOST
          value: "$(Get-Content -Path env:K8S_TO_POSTGRESQL)"
        - name: DB_PORT
          value: "5432"
        - name: DB_NAME
          value: "postgres"
"@
$DeployDynamicExampleapp | kubectl create -f -

  10. Verify the pod is in a running state.

    $ kubectl get pods --namespace dynamic-exampleapp
NAME                                    READY   STATUS    RESTARTS   AGE
hashicups-exampleapp-7cfb565cd5-vkbrx   1/1     Running   0          37s

  11. Access the applications secret endpoint to display the Kubernetes Secret created by VSO.

    kubectl exec --namespace dynamic-exampleapp --stdin=true `
  $(kubectl get pods --namespace dynamic-exampleapp -l app=hashicups-exampleapp -o name) `
  -- curl http://127.0.0.1:5000/secret --silent

    Example output:

    {"password":"CWZODcdw-Di97dVWzRgd","username":"v-kubernet-vault-ro-W4Y577NGFzE7gJgdqXuW-1726076758"}

  12. Access the applications status endpoint to verify the pod can connect to the PostgreSQL database.

    kubectl exec --namespace dynamic-exampleapp --stdin=true `
  $(kubectl get pods --namespace dynamic-exampleapp -l app=hashicups-exampleapp -o name) `
  -- curl http://127.0.0.1:5000/status --silent

    Example output:

    {"message":"Connected to the database successfully","status":"success"}

    The app connected to the PostgreSQL database using the dynamic secrets created by Vault, and synchronized to a Kubernetes Secret by VSO.

Set up persistent client cache

By default, the Vault client cache does not persist. This can lead to VSO not maintaining dynamic secret leases through restarts and upgrades of the Vault Secrets Operator.

The client cache enables seamless upgrades because Vault can track and renew tokens and dynamic secret leases through leadership changes.

Client cache persistence and encryption is not enabled by default because it requires extra configuration and Vault server setup.

  1. Enable the transit secrets engine.

    $ vault secrets enable  transit
Success! Enabled the transit secrets engine at: transit/

  2. Create an encryption key.

    $ vault write -force transit/keys/vso-client-cache
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1726154823]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      vso-client-cache
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

  3. Create a policy that allows VSO to access the transit secrets engine.

    $VaultTransitPolicy = @"
path "transit/encrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
path "transit/decrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
"@
$VaultTransitPolicy | vault policy write vso-transit-policy -

    Example output:

    Success! Uploaded policy: vso-transit-policy

  4. Create a role in the Kubernetes auth method used by VSO to authenticate with Vault and attach the vso-transit-policy.

    vault write auth/kubernetes/role/operator `
bound_service_account_names=vault-secrets-operator-controller-manager `
bound_service_account_namespaces=vault-secrets-operator `
token_period="24h" `
token_policies=vso-transit-policy

    Example output:

    Success! Data written to: auth/kubernetes/role/operator

    The vault-secrets-operator-controller-manager service account and vault-secrets-operator are the defaults created when deploying VSO.

  5. Create a file to update the VSO Helm deployment.

    $UpdateVSOHelm = @"
controller:
  manager:
    clientCache:
      persistenceModel: direct-encrypted
      storageEncryption:
        enabled: true
        vaultConnectionRef: vault/vault-connection
        keyName: vso-client-cache
        transitMount: transit
        method: kubernetes
        mount: kubernetes
        kubernetes:
          role: operator
          serviceAccount: vault-secrets-operator-controller-manager
"@
$UpdateVSOHelm | Set-Content -Path .\update-values.yml

  6. Upgrade the VSO deployment to include the secrets cache.

    helm upgrade -f update-values.yml vault-secrets-operator hashicorp/vault-secrets-operator `
  --namespace vault-secrets-operator

    Example output:

    Release "vault-secrets-operator" has been upgraded. Happy Helming!
NAME: vault-secrets-operator
LAST DEPLOYED: Thu Sep 12 11:30:37 2024
NAMESPACE: vault-secrets-operator
STATUS: deployed
REVISION: 2

  7. Check the vault-secrets-operator namespace for the encrypted client cache.

    $ kubectl get secrets -n vault-secrets-operator
sh.helm.release.v1.vault-secrets-operator.v1   helm.sh/release.v1   1      44h
sh.helm.release.v1.vault-secrets-operator.v2   helm.sh/release.v1   1      53s
vso-cc-kubernetes-a1fd22e2726ce4c715a25f       Opaque               2      20s
vso-cc-kubernetes-e086c5f655898e3544b9b8       Opaque               2      20s
vso-cc-storage-hmac-key                        Opaque               1      44h

VSO stores the client cache as a Kubernetes Secret prefixed with vso-cc in the vault-secrets-operator namespace.

In this example, there is a cache for both Kubernetes namespaces VSO is managing secrets for - static-exampleapp and dynamic-exampleapp.

Set up instant updates with Vault Enterprise

Tip

The instant updates option requires Vault Enterprise 1.16.3+ due to the use of Vault event notifications.

There are times when secrets updated in Vault need to be immediately available to applications to avoid downtime. To ensure the timely update of secrets by the Vault Secrets Operator, you can use Vault event notifications.

Instant update supports the Vault KV v2 secrets engine, and the Kubernetes VaultStaticSecret resource type.

  1. Update the Vault exampleapp-kv-read policy to include read access to events, as well as list and subscribe on the exampleapp-kv-read policy.

    $InstantUpdatePolicy = @"
path "static/data/exampleapp/creds" {
  capabilities = ["read", "list", "subscribe"]
  subscribe_event_types = ["*"]
}

path "sys/events/subscribe/kv*" {
  capabilities = ["read"]
}
"@
$InstantUpdatePolicy | vault policy write exampleapp-kv-read -

    Example output:

    Success! Uploaded policy: exampleapp-kv-read

  2. Add syncConfig.instantUpdates=true to the vault-static-secret resource in the static-exampleapp namespace.

    $StaticSecretInstantUpdate = @"
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-static-secret
  namespace: static-exampleapp
spec:
  vaultAuthRef: vault-auth-vso-static-exampleapp
  mount: static
  type: kv-v2
  path:  exampleapp/creds
# version: 2
  refreshAfter: 300s
  destination:
    create: true
    name: vso-static-creds-from-vault
  syncConfig:
    instantUpdates: true
---
"@
$StaticSecretInstantUpdate | kubectl apply -f -

    Note

    It is okay to proceed if you receive the following error message:

    Warning: resource vaultstaticsecrets/vault-static-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

    Example output:

    vaultstaticsecret.secrets.hashicorp.com/vault-static-secret configured

  3. Verify the VSO configuration is now watching for events.

    $ kubectl describe vaultstaticsecret vault-static-secret --namespace static-exampleapp
...snip...
Events:
Type    Reason               Age   From               Message
----    ------               ----  ----               -------
Normal  SecretRotated        110s  VaultStaticSecret  Secret synced
Normal  EventWatcherStarted  110s  VaultStaticSecret  Started watching events

  4. Update the /static/exampleapp/creds secret in Vault.

    vault kv put static/exampleapp/creds `
 username='jalbertson' `
 password='bestpasswordever'

    Example output:

    ======== Secret Path ========
static/data/exampleapp/creds

======= Metadata =======
Key                Value
---                -----
created_time       2024-10-11T17:25:40.390838793Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

  5. Verify the Kubernetes Secret values match the updated Vault secret value.

    $ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"data\":{\"password\":\"bestpasswordever\",\"username\":\"jalbertson\"},\"metadata\":{\"created_time\":\"2024-09-30T19:17:39.666589844Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":2}}",
  "password": "bestpasswordever",
  "username": "jalbertson"
}

    Updating the secret value triggered an event in Vault, which initiated updating the Kubernetes Secret.

Summary

In this lab, you deployed the Vault Secrets Operator and configured VSO to work with an external Vault cluster. You then configured VSO to read both static and dynamic secrets and create Kubernetes native secrets in multiple Kubernetes namespaces. Finally, you deployed the application and validated it can read the secrets created by VSO by connecting the application to the PostgreSQL database.

Now that the app is set up in support of the new smart brewing machines, Alice and Watson head to that famous Boston donut shop to enjoy a well deserved iced coffee (Yea, I know, iced coffee in October? It's a Boston thing!)!

Clean up

  1. Stop the Vault Enterprise, and PostgreSQL container.

    $ docker kill vault-enterprise learn-postgres

  2. Stop minikube.

    $ minikube stop

  3. (Optional) Delete the minikube instance.

    $ minikube delete

  4. Return to the terminal running the Kubernetes proxy and type ctrl-c to stop the proxy.

  5. Return to the terminal running the ngrok type ctrl-c to stop ngrok.

  6. Delete the HC_LAB directory and lab content.

    Remove-Item -Path ~/HC_LAB -Force

Help and reference

Previous

Manage Kubernetes native secrets

 Next Collection

Authentication

Operating system