Retrieve secrets for Kubernetes workloads with Vault Agent

13min
|
Vault

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

Automatic authentication
Secure delivery/storage of tokens
Lifecycle management of these tokens (renewal & re-authentication)

Note

Though this tutorial focuses on the demonstration of the Vault Agent Auto-Auth using the kubernetes auth method for application integration, there are two other options. The other options are the Vault Secrets Operator and the Agent Sidecar Injector.

Refer to Vault Secrets Operator and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar for a step-by-step tutorial on these options.

Prerequisites

To perform the tasks described in this tutorial, you need:

Docker
Kubernetes command-line interface (CLI)
minikube
A running Vault environment reachable from your Kubernetes environment. Refer to the Vault install guide to install Vault. Make sure that your Vault server has been initialized and unsealed

Install supporting tools

This tutorial was last tested 13 November 2023 on a macOS 13.5.2 using the following software versions.

Docker version

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
...

Kubectl version

$ kubectl version --short
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
...

minikube version

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.
```
$ git clone https://github.com/hashicorp-education/learn-vault-agent
```
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.
Go into the learn-vault-agent/vault-agent-k8s-demo directory.
```
$ cd learn-vault-agent/vault-agent-k8s-demo
```

Working directory

This tutorial assumes that the remainder of commands are executed in this directory.

Start a Vault server

To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.
```
$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
```
Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://0.0.0.0:8200
```

Create a service account

Start a Kubernetes cluster running in minikube.
```
$ minikube start --driver=docker
```

Wait a couple of minutes for the minikube environment to become fully available.

$ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided vault-auth-service-account.yaml file in your preferred text editor and examine its content for the service account definition to be used for this tutorial.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

Create the vault-auth service account.

$ kubectl apply --filename vault-auth-service-account.yaml

Kubernetes 1.24+ only

The service account generated a secret that is required for configuration automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.

apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

Create a vault-auth-secret secret.

$ kubectl apply --filename vault-auth-secret.yaml

Configure Kubernetes auth method

Create a read-only policy, myapp-kv-ro in Vault.

$ vault policy write myapp-kv-ro - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

Create some test data at the secret/myapp path.

$ vault kv put secret/myapp/config \
      username='appuser' \
      password='suP3rsec(et!' \
      ttl='30s'

Output:

====== Secret Path ======
secret/data/myapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-24T06:09:49.99472Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Set the environment variables to point to the running minikube environment.

Set the SA_SECRET_NAME environment variable value to the vault-auth service account secret.

$ export SA_SECRET_NAME=$(kubectl get secrets --output=json \
    | jq -r '.items[].metadata | select(.name|startswith("vault-auth-")).name')

Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

$ export SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME \
    --output 'go-template={{ .data.token }}' | base64 --decode)

Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

$ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode | awk '{printf "%s\\n", $0}')

Set the K8S_HOST environment variable value to minikube IP address.

$ export K8S_HOST=$(kubectl config view --raw --minify --flatten \
    --output 'jsonpath={.clusters[].cluster.server}')

Now, enable and configure the Kubernetes auth method.

Enable the Kubernetes auth method at the default path ("auth/kubernetes").

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Tell Vault how to communicate with the Kubernetes (minikube) cluster.

$ vault write auth/kubernetes/config \
     token_reviewer_jwt="$SA_JWT_TOKEN" \
     kubernetes_host="$K8S_HOST" \
     kubernetes_ca_cert="$SA_CA_CRT" \
     issuer="https://kubernetes.default.svc.cluster.local"

Output:

Success! Data written to: auth/kubernetes/config

You can validate the issuer name of your Kubernetes cluster found under Discovering the service account issuer.

Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

$ vault write auth/kubernetes/role/example \
     bound_service_account_names=vault-auth \
     bound_service_account_namespaces=default \
     token_policies=myapp-kv-ro \
     ttl=24h

Output:

Success! Data written to: auth/kubernetes/role/example

Note

The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.

Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

Start a minikube SSH session.

$ minikube ssh
## ... minikube ssh login

Within this SSH session, retrieve the value of the minikube host. TODO: is there a better way?
```
$ dig +short host.docker.internal
192.168.65.2
```
Docker networking
The host has a changing IP address (or none if you have no network access). It is recommended that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.

Next, retrieve the status of the Vault server to verify network connectivity.

$ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status | python3 -m json.tool
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.4.1+ent",
  "migration": false,
  "cluster_name": "vault-cluster-3de6c2d3",
  "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
  "recovery_seal": false,
  "storage_type": "inmem"
}

The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

Next, exit the minikube SSH session.
```
$ exit
```

Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the minikube gateway address.

$ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

Verify that the variable contains the IP address you saw when executed in the minikube shell.
```
$ echo $EXTERNAL_VAULT_ADDR
192.168.65.2
```

Optional: Verify the Kubernetes auth method configuration

Define a Pod with a container.

$ cat > devwebapp.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  serviceAccountName: vault-auth
  containers:
    - name: devwebapp
      image: burtlo/devwebapp-ruby:k8s
      env:
        - name: VAULT_ADDR
          value: "http://$EXTERNAL_VAULT_ADDR:8200"
EOF

The Pod is named devwebapp and runs with the vault-auth service account.

Create the devwebapp pod in the default namespace

$ kubectl apply --filename devwebapp.yaml --namespace default
pod/devwebapp created

Display all the pods in the default namespace. Wait until the devwebapp pod is running and ready (1/1).

$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
devwebapp   1/1     Running   0          77s

Start an interactive shell session on the devwebapp pod.
```
$ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh
#
```
Your system prompt is replaced with a new prompt #.
Note
The prompt within this section is shown as $ but the commands are intended to be executed within this interactive shell on the devwebapp container.

Set KUBE_TOKEN to the service account token.

$ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Authenticate with Vault through the example role with the KUBE_TOKEN.

$ curl --request POST \
       --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
       $VAULT_ADDR/v1/auth/kubernetes/login | python3 -m json.tool

Example output:

{
   ...snip...
   "auth": {
      "client_token": "hvs.CAESIBxfvzds7M4tas017ls36Dl_kA-3YpCCBT9wczP1E41DGh4KHGh2cy5JYW1NMmFkb2gwTVBhZVhsN0pDM0tOaWM",
      "accessor": "1Miu5tbfuZzYuYWCntb4Ztke",
      "policies": [
         "default",
         "myapp-kv-ro"
      ],
      "token_policies": [
         "default",
         "myapp-kv-ro"
      ],
      "metadata": {
         "role": "example",
         "service_account_name": "vault-auth",
         "service_account_namespace": "default",
         "service_account_secret_name": "",
         "service_account_uid": "a293b5fb-96a2-43b0-acee-03d9ffce9423"
      },
      "lease_duration": 86400,
      "renewable": true,
      "entity_id": "8925670b-d8d6-0792-ee56-eec1a9e740cb",
      "token_type": "service",
      "orphan": true,
      "mfa_requirement": null,
      "num_uses": 0
   }
}

Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

Lastly, exit the pod.
```
$ exit
```

Start Vault Agent with Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

First, open the provided configmap.yaml file in your preferred text editor and review its content.

apiVersion: v1
data:
  vault-agent-config.hcl: |
    # Comment this out if running as sidecar instead of initContainer
    exit_after_auth = true

    pid_file = "/home/vault/pidfile"

    auto_auth {
        method "kubernetes" {
            mount_path = "auth/kubernetes"
            config = {
                role = "example"
            }
        }

        sink "file" {
            config = {
                path = "/home/vault/.vault-token"
            }
        }
    }

    template {
    destination = "/etc/secrets/index.html"
    contents = <<EOT
    <html>
    <body>
    <p>Some secrets:</p>
    {{- with secret "secret/data/myapp/config" }}
    <ul>
    <li><pre>username: {{ .Data.data.username }}</pre></li>
    <li><pre>password: {{ .Data.data.password }}</pre></li>
    </ul>
    {{ end }}
    </body>
    </html>
    EOT
    }
kind: ConfigMap
metadata:
  name: example-vault-agent-config
  namespace: default

This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Configure Kubernetes auth method.

The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

The template block creates a file which retrieves username and password values at the secret/data/myapp/config path.

Create a ConfigMap containing a Vault Agent configuration.

$ kubectl create --filename configmap.yaml
configmap/example-vault-agent-config created

View the created ConfigMap.

$ kubectl get configmap example-vault-agent-config --output yaml

An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-auth

  volumes:
    - configMap:
        items:
          - key: vault-agent-config.hcl
            path: vault-agent-config.hcl
        name: example-vault-agent-config
      name: config
    - emptyDir: {}
      name: shared-data

  initContainers:
    - args:
        - agent
        - -config=/etc/vault/vault-agent-config.hcl
        - -log-level=debug
      env:
        - name: VAULT_ADDR
          value: http://EXTERNAL_VAULT_ADDR:8200
      image: vault
      name: vault-agent
      volumeMounts:
        - mountPath: /etc/vault
          name: config
        - mountPath: /etc/secrets
          name: shared-data

  containers:
    - image: nginx
      name: nginx-container
      ports:
        - containerPort: 80
      volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: shared-data

The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example pod. A Vault container which runs Vault Agent as an Init Container and an nginx container exposing port 80.

The Vault address, VAULT_ADDR, is set to a placeholder value EXTERNAL_VAULT_ADDR.

Generate the Pod spec with EXTERNAL_VAULT_ADDR variable value in its place.

$ cat example-k8s-spec.yaml | \
   sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \
   > vault-agent-example.yaml

Create the vault-agent-example pod defined in vault-agent-example.yaml.
```
$ kubectl apply --filename vault-agent-example.yaml --record
```
This takes a minute or so for the pod to become fully up and running.

Verification

In another terminal, launch the minikube dashboard.
```
$ minikube dashboard
```
Under Workloads click Pods to verify that vault-agent-example pod has been created successfully.
Select vault-agent-example to see its details.
In another terminal, port forward all requests made to http://localhost:8080 to port 80 on the vault-agent-example pod.
```
$ kubectl port-forward pod/vault-agent-example 8080:80
```
In a web browser, go to localhost:8080
Notice that the username and password values were successfully read from secret/myapp/config.

Optionally, you can view the HTML source.

$ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html
<html>
<body>
<p>Some secrets:</p>
<ul>
<li><pre>username: appuser</pre></li>
<li><pre>password: suP3rsec(et!</pre></li>
</ul>

</body>
</html>

Clean up

Stop the cluster.

$ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

Delete the cluster.

$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/mrken/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster

Delete the supporting code repository copied locally.
```
$ cd ../.. && rm -rf learn-vault-agent
```
Stop the Vault server process.
```
$ pkill vault
```

Help and reference

This tutorial also appears in:

7 tutorials

Vault Agent
Use Vault Agent to authenticate and read secrets from Vault with little to no change in your application code.
- Vault

Challenge

Solution

Prerequisites

Install supporting tools

Start a Vault server

Create a service account

Kubernetes 1.24+ only

Configure Kubernetes auth method

Determine the Vault address

Optional: Verify the Kubernetes auth method configuration

Start Vault Agent with Auto-Auth

Verification

Clean up

Help and reference

Other Vault Agent tutorials:

This tutorial also appears in: