HCP Vault Dedicated with Amazon Elastic Kubernetes Service

11min
|
HCP
Vault

HashiCorp Cloud Platform (HCP) is a fully managed platform offering HashiCorp Products as a Service (HPaaS) to automate infrastructure on any cloud.

This tutorial will cover the process required to connect an Elastic Kubernetes Service (EKS) Cluster to HCP Vault Dedicated on AWS.

Prerequisites

The following prerequisites are required:

An HCP HashiCorp Virtual Network (HVN)
A public Vault Dedicated deployment
AWS CLI
kubectl
helm
jq
git
An EKS Cluster version 1.24 or higher deployed in the VPC associated with your HVN

For this tutorial, you will need to ensure that you have authenticated with the AWS CLI, and that the CLI is targeting the region where you have created your EKS cluster. Review the AWS documentation for instructions on how to configure the AWS CLI.

To ensure that communication between your Vault Dedicated cluster servers and the agent running in your EKS cluster is possible, you must complete the steps detailed in the manual deployment tutorial.

Your EKS cluster security group must allow traffic from the Vault Dedicated CIDR range. If your EKS cluster endpoint uses port 443, create a security group rule to allow ingress traffic from Vault Dedicated to the primary EKS cluster security group.

For this tutorial, you will configure Vault from your development host. As a result, the Vault Dedicated cluster needs to be publicly available. In production, you should configure Vault Dedicated over a bastion host.

Configure development host

Kubernetes stores cluster connection information in a special file called kubeconfig. You can retrieve the Kubernetes configuration settings for your EKS cluster and merge them into your local kubeconfig file.

Use the AWS CLI to retrieve the kubeconfig.

$ aws eks --region <your-region> update-kubeconfig --name <your-cluster-name>

You can use the HCP Portal to retrieve the client configuration information you need to connect the Vault agents in your EKS cluster to Vault Dedicated. Navigate to the Vault resource page in the HCP portal, and then select the Vault cluster.
Click "Generate Token". Copy the administrator token and set it in your terminal to an environment variable named VAULT_TOKEN.
```
$ export VAULT_TOKEN=s.*******************
```
Click the clipboard next to "Public". This will copy the public Vault address to your clipboard. Set it as the VAULT_ADDR environment variable.
```
$ export VAULT_ADDR=https://learn.vault.**************.aws.hashicorp.cloud:8200
```
Click the clipboard next to "Private". This will copy the private Vault address to your clipboard. Set the Vault private address as the VAULT_PRIVATE_ADDR environment variable. You will use this later in the tutorial to enable the EKS cluster to access Vault Dedicated over the HVN peering connection.
```
$ export VAULT_PRIVATE_ADDR=https://learn.private.vault.**************.aws.hashicorp.cloud:8200
```
Since Vault Dedicated uses namespaces, set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```

Install Vault agents on EKS

This uses the official vault-helm chart to install the Vault agents to your EKS cluster.

Retrieve the Helm chart from the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

Example output:

"hashicorp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈ Happy Helming!⎈

Create a values.yaml file that sets the external servers to Vault Dedicated. This will deploy a Vault agent injector into the EKS cluster.
```
$ cat > values.yaml << EOF
injector:
   enabled: true
   externalVaultAddr: "${VAULT_PRIVATE_ADDR}"
EOF
```
Validate that the values file is populated correctly.
```
$ more values.yaml
```

Install the HashiCorp Vault Helm chart.

$ helm install vault -f values.yaml hashicorp/vault

Once the helm install command completes, verify the Vault agent injector pod deploys by issuing kubectl get pods.

$ kubectl get pods

Example output:

NAME                                   READY   STATUS    RESTARTS   AGE
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m11s

Configure Kubernetes auth method on HCP Vault Dedicated

Your services need a Kubernetes service account token to authenticate to Vault.

Enable the Kubernetes auth method in Vault Dedicated using your terminal.

$ vault auth enable kubernetes

Output:

Success! Enabled kubernetes auth method at: kubernetes/

The chart configures a Kubernetes service account named vault that you can use to enable Vault communication with Kubernetes.
View the available service accounts.
```
$ kubectl get serviceaccounts | grep vault

vault                  0         5m16s
vault-agent-injector   0         5m16s
```

Create a token for the vault service account.

Note

As of Kubernetes 1.24 service account tokens are not created automatically.

$ kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  annotations:
    kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
EOF

Example output:

secret/vault-token created

Get the JSON Web Token (JWT) for the vault service account and set it to the TOKEN_REVIEW_JWT environment variable.

$ export TOKEN_REVIEW_JWT=$(kubectl get secret vault-token -o json \
  | jq -r '.data | .token' \
  | base64 --decode)

Get the Kubernetes certificate authority for the service account and set it to the KUBE_CA_CERT environment variable.

$ export KUBE_CA_CERT=$(kubectl get secret vault-token -o json \
  | jq -r '.data | ."ca.crt"' \
  | base64 --decode)

Get the Kubernetes cluster endpoint and set it to the KUBE_HOST environment variable.

$ export KUBE_HOST=$(kubectl config view --raw --minify --flatten \
   -o jsonpath='{.clusters[].cluster.server}')

Configure the Vault Kubernetes auth method to use the service account token.

$ vault write auth/kubernetes/config \
   token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
   kubernetes_host="$KUBE_HOST" \
   kubernetes_ca_cert="$KUBE_CA_CERT"

Output:

Success! Data written to: auth/kubernetes/config

Deploy an example workload

Now that the clients have been deployed, it is time to deploy an application workload. This tutorial will use the HashiCups demo application.

Issue the following command to clone the repository to the development host.
```
$ git clone https://github.com/hashicorp/vault-guides.git
```
Change directory into the example repository.
```
cd vault-guides/cloud/eks-hcp-vault/
```

Deploy a database to Kubernetes

Note

This tutorial deploys a database into Kubernetes and exposes it to HCP Vault Dedicated using a public LoadBalancer service type. In a production configuration, you will want to deploy this as a private load balancer restricting access to Vault Dedicated.

Deploy a PostgreSQL database. This contains data for various coffees related to a demo application, all hosted in the products database.
```
$ kubectl apply -f postgres.yaml
```
Output:
```
service/postgres created
deployment.apps/postgres created
```

Verify you've deployed the PostgreSQL database in your Kubernetes cluster.

$ kubectl get pods

Example output:

NAME                                   READY   STATUS    RESTARTS   AGE
postgres-5bbfc8bb5c-9px77              1/1     Running   0          31s
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m34s

Add the database role to Vault

The product API needs to read the database username and password from Vault. Create the role for the product service account to generate database credentials.

Enable the database secrets engine.

$ vault secrets enable database

Output:

Success! Enabled the database secrets engine at: database/

Set the POSTGRES_IP environment variable to the load balancer DNS hostname.

$ export POSTGRES_IP=$(kubectl get service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
   postgres)

Create the database configuration that allows Vault Dedicated to configure Postgres.

$ vault write database/config/products \
    plugin_name=postgresql-database-plugin \
    allowed_roles="*" \
    connection_url="postgresql://{{username}}:{{password}}@${POSTGRES_IP}:5432/products?sslmode=disable" \
    username="postgres" \
    password="password"

Create a database role for product that allows Vault to issue database passwords.

$ vault write database/roles/product \
    db_name=products \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\
    default_ttl="1h" \
    max_ttl="24h"

Output:

Success! Data written to: database/roles/product

Request a new set of PostgreSQL database credentials for the product role.

$ vault read database/creds/product

Example output:

Key                Value
---                -----
lease_id           database/creds/product/zeXVyjlT550wHYEDOhF5ckDC.8fmet
lease_duration     1h
lease_renewable    true
password           gT8FvhyMtLuF-SCbw4mF
username           v-token-hc-product-Qcg6fDERVvya93uwvbCy-1652888425

Configure Vault policy for database credentials

Create a file called product.hcl that allows the product service to read the database credentials specific to product.
```
$ cat > product.hcl << EOF
path "database/creds/product" {
  capabilities = ["read"]
}
EOF
```

Create a new product policy.

$ vault policy write product ./product.hcl

Output:

Success! Uploaded policy: product

Configure Vault to associate the product service with a Kubernetes service account. This allows the product service account in Kubernetes to get a Vault token.

$ vault write auth/kubernetes/role/product \
    bound_service_account_names=product \
    bound_service_account_namespaces=default \
    policies=product \
    ttl=1h

Output:

Success! Data written to: auth/kubernetes/role/product

Deploy the product API

Make sure that you include your Vault Dedicated namespace in the deployment that will access database credentials.
```
$ grep ${VAULT_NAMESPACE} product.yaml
```
Output:
```
vault.hashicorp.com/namespace: "admin"
```

Deploy the product service.

$ kubectl apply -f product.yaml

Output:

service/product created
serviceaccount/product created
deployment.apps/product created

The product deployment should initialize.

$ kubectl get pods

Example output:

NAME                                    READY   STATUS    RESTARTS   AGE
postgres-5bd8c648fd-q6jhr               1/1     Running   0          25m
product-6f7ff64c75-zrcwd                2/2     Running   0          27s
vault-agent-injector-7d4ccf785f-dkt7c   1/1     Running   0          29m

Port forward the web service locally to port 9090.

$ kubectl port-forward service/product 9090

Output:

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Open another terminal and make a request to localhost:9090/coffees to check if the web service can pull coffee information from the database.

$ curl -s localhost:9090/coffees | jq .

Output:

[
  {
    "id": 1,
    "name": "Packer Spiced Latte",
    "teaser": "Packed with goodness to spice up your images",
    "description": "",
    "price": 350,
    "image": "/packer.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      },
      {
        "ingredient_id": 4
      }
    ]
  },
  {
    "id": 2,
    "name": "Vaulatte",
    "teaser": "Nothing gives you a safe and secure feeling like a Vaulatte",
    "description": "",
    "price": 200,
    "image": "/vault.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      }
    ]
  },
  {
    "id": 3,
    "name": "Nomadicano",
    "teaser": "Drink one today and you will want to schedule another",
    "description": "",
    "price": 150,
    "image": "/nomad.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 3
      }
    ]
  },
  {
    "id": 4,
    "name": "Terraspresso",
    "teaser": "Nothing kickstarts your day like a provision of Terraspresso",
    "description": "",
    "price": 150,
    "image": "/terraform.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 5,
    "name": "Vagrante espresso",
    "teaser": "Stdin is not a tty",
    "description": "",
    "price": 200,
    "image": "/vagrant.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 6,
    "name": "Connectaccino",
    "teaser": "Discover the wonders of our meshy service",
    "description": "",
    "price": 250,
    "image": "/consul.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 5
      }
    ]
  }
]

When you are done, return to the terminal with the port-forward command and type Ctrl + C to exit.

Clean up

Delete the product API.
```
$  kubectl delete -f product.yaml
```

Delete the product role.

$ vault delete auth/kubernetes/role/product

Delete the product policy.
```
$ vault policy delete product
```
Delete the product database role.
```
$ vault delete database/roles/product
```

Revoke all leases for database credentials.

$ vault lease revoke -prefix database/creds/product

Delete the PostgreSQL database.
```
$ kubectl delete -f postgres.yaml
```
Delete the database secrets engine configuration.
```
$ vault delete database/config/product
```
Delete the Helm installation for Vault Dedicated.
```
$ helm delete vault
```
Disable the database secrets engine in Vault Dedicated.
```
$ vault secrets disable database
```
Disable the Kubernetes auth method in Vault Dedicated.
```
$ vault auth disable kubernetes
```
Delete Vault Dedicated and the HVN.

Next steps

In this tutorial, you connected Vault clients on Amazon EKS to Vault Dedicated and retrieved PostgreSQL database credential dynamically. To learn more about the Vault features introduced in this tutorial, refer to the following tutorials.

Deploy the Kubernetes VSO with HCP Vault

HCP Vault with AWS EKS and JWT auth method