Codify management of HCP Vault Dedicated

Challenge

Manual system administration can become a challenge as the scale of infrastructure increases. Often, an organization must manage multiple Vault environments (development, testing, staging, production, etc.). Keeping up with the increasing management demand soon becomes a challenge without some sort of automation.

Solution

One of the pillars behind the Tao of Hashicorp is automation through codification.

HashiCorp Terraform is an infrastructure as code software tool that enables operation teams to codify Vault configuration tasks such as the creation of policies. Automation through codification allows operators to increase their productivity, move quicker, promote repeatable processes, and reduce human error.

This tutorial demonstrates techniques for creating Vault policies and configurations using the Terraform Vault Provider.

Prerequisites

• Terraform installed
• HCP Vault Dedicated cluster with public cluster address enabled

HCP Vault Dedicated cluster

For this tutorial, follow the enable the public cluster address enabled in the Deploy HCP Vault Dedicated with Terraform tutorial.

resource "hcp_hvn" "learn_hcp_vault_hvn" {
  hvn_id         = var.hvn_id
  cloud_provider = var.cloud_provider
  region         = var.region
}

resource "hcp_vault_cluster" "learn_hcp_vault" {
  hvn_id     = hcp_hvn.learn_hcp_vault_hvn.hvn_id
  cluster_id = var.cluster_id
  public_endpoint = true
}

Scenario introduction

Vault administrators must manage multiple Vault environments. To automate the Vault server configuration, you are going to use Terraform to provision the following Vault resources.

For an HCP Vault Dedicated cluster, the root level namespace is admin. In this tutorial, you are going to create nested namespaces as shown below.

admin
 ├── education
 │   └── training
 │       └── boundary
 └── test

Terraform will provision the following Vault resources.

Examine the Terraform files

Clone or download the demo assets from the hashicorp-education/learn-vault-hcp-codify-mgmt GitHub repository to perform the steps described in this tutorial.

To clone the repository, use the git clone command.

$ git clone https://github.com/hashicorp-education/learn-vault-hcp-codify-mgmt.git

Alternatively, you can download the repository.

This repository contains supporting content for all of the Vault tutorials. The content specific to this tutorial can be found within a sub-directory.

Change the working directory to learn-vault-hcp-codify-mgmt.

$ cd learn-vault-hcp-codify-mgmt

The directory contains Terraform files to configure Vault.

$ tree
.
├── auth.tf
├── main.tf
├── policies
│   ├── admin-policy.hcl
│   ├── eaas-client-policy.hcl
│   └── tester.hcl
├── policies.tf
└── secrets.tf

Review main.tf

Open the main.tf file in your preferred text editor to examine its content. It defines three vault_namespace blocks each pointing to a different namespace.

#------------------------------------------------------------------------------
# The best practice is to use remote state file and encrypt it since your
# state files may contains sensitive data (secrets).
#------------------------------------------------------------------------------
# terraform {
#       backend "s3" {
#             bucket = "remote-terraform-state-dev"
#             encrypt = true
#             key = "terraform.tfstate"
#             region = "us-east-1"
#       }
# }


#------------------------------------------------------------------------------
# To leverage more than one namespace, define a vault provider per namespace
#
#   admin
#    ├── education
#    │   └── training
#    │       └── boundary
#    └── test
#------------------------------------------------------------------------------
terraform {
  required_providers {
    vault = "~> 3.8.0"
  }
}


provider "vault" {}

#--------------------------------------
# Create 'admin/education' namespace
#--------------------------------------
resource "vault_namespace" "education" {
  path = "education"
}

#---------------------------------------------------
# Create 'admin/education/training' namespace
#---------------------------------------------------
resource "vault_namespace" "training" {
  namespace  = vault_namespace.education.path
  path       = "training"
}

#-----------------------------------------------------------
# Create 'admin/education/training/boundary' namespace
#-----------------------------------------------------------
resource "vault_namespace" "boundary" {
  namespace  = vault_namespace.training.path_fq
  path       = "boundary"
}

#--------------------------------------
# Create 'admin/test' namespace
#--------------------------------------
resource "vault_namespace" "test" {
  path = "test"
}

To create a child namespace education under admin, you need to do nothing as all namespaces are by default created under admin.

To create a sub-namespace under education, add a namespace field with reference to the parent namespace.

resource "vault_namespace" "training" {
  namespace = vault_namespace.education.path
  path = "training"
}

Review policies.tf

Open the policies.tf file and examine the vault_policy resources.

# Create an admins policy in the admin namespace
resource "vault_policy" "admin_policy" {
  namespace = vault_namespace.test.path
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# # Create an admins policy in the admin/education namespace
resource "vault_policy" "admin_policy_education" {
  namespace = vault_namespace.education.path
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# # Create an admins policy in the admin/education/training namespace
resource "vault_policy" "admin_policy_training" {
  namespace = vault_namespace.training.path_fq
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admins policy in the admin/education/training/boundary namespace
resource "vault_policy" "admin_policy_boundary" {
  namespace = vault_namespace.boundary.path_fq
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create an admins policy in the admin/test namespace
resource "vault_policy" "admin_policy_test" {
  namespace = vault_namespace.test.path
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create a tester policy in the admin/test namespace
resource "vault_policy" "tester_policy" {
  namespace = vault_namespace.test.path
  name   = "tester"
  policy = file("policies/tester.hcl")
}

# Create an eaas-client policy in the admin/education namespace
resource "vault_policy" "eaas-client_policy" {
  namespace = vault_namespace.education.path
  name   = "eaas-client"
  policy = file("policies/eaas-client-policy.hcl")
}

This creates an admins policy in all namespaces, an eaas-client policy in the admin/education namespace (line 33 through 38), and a tester policy in the admin/test namespace (line 41 through 46).

Review auth.tf

Open the auth.tf file. This enables userpass auth method in the admin/test namespace and creates a user, "student" with tester policy attached. The password is set to "changeme".

#------------------------------------------------------------
# Enable userpass auth method in the 'admin/test' namespace
#------------------------------------------------------------
resource "vault_auth_backend" "userpass" {
  namespace = vault_namespace.test.path
  type = "userpass"
}

#-----------------------------------------------------------
# Create a user named 'student' with password, 'changeme'
#-----------------------------------------------------------
resource "vault_generic_endpoint" "student" {
  path = join("/", ["auth", vault_auth_backend.userpass.path,"users/student"])
  namespace = vault_namespace.test.path
  ignore_absent_fields = true

  data_json = <<EOT
{
  "policies": ["tester"],
  "password": "changeme"
}
EOT
}

Review secrets.tf

Open the secrets.tf file.

# Enable kv-v2 secrets engine in the education namespace
resource "vault_mount" "kv-v2" {
  namespace = vault_namespace.education.path
  path = "kv-v2"
  type = "kv-v2"
}

# Enable kv-v2 secrets engine in the 'admin/test' namespace at 'secret' path
resource "vault_mount" "secret" {
  namespace = vault_namespace.test.path
  path = "secret"
  type = "kv-v2"
}

# Enable Transit secrets engine at 'transit' in the 'admin/education' namespace
resource "vault_mount" "transit" {
  namespace = vault_namespace.education.path
  path = "transit"
  type = "transit"
}

# # Creating an encryption key named 'payment'
resource "vault_transit_secret_backend_key" "key" {
  namespace = vault_namespace.education.path
  backend    = vault_mount.transit.path
  name       = "payment"
  deletion_allowed = true
}

This Terraform file performs the following:

• Line 2-7: Enables kv-v2 secrets engine at kv-v2 in the admin/education namespace.
• Line 10-15: Enables kv-v2 secrets engine at secret in the admin/test namespace.
• Line 18-23: Enables transit secrets engine in the admin/education namespace.
• Line 26-32: Creates payment key

Run Terraform to configure Vault

1. Launch the HCP Portal and login.

2. Click Vault in the left navigation pane.

3. In the Vault clusters pane, click vault-cluster.

4. Under Cluster URLs, click Public Cluster URL.

5. In a terminal, set the VAULT_ADDR environment variable to the copied address.

   $ export VAULT_ADDR=<Public_Cluster_URL>
   

6. Return to the Overview page and click Generate token.

   Within a few moments, a new token will be generated.

7. Copy the Admin Token.

8. Return to the terminal and set the VAULT_TOKEN environment variable.

   $ export VAULT_TOKEN=<token>
   

9. Initialize Terraform to pull Vault provider plugin.

   $ terraform init
   

   This downloads the Vault plugin. When it completes, it displays a message, Terraform has been successfully initialized!

10. Execute the apply command to configure Vault.

    $ terraform apply
    

    This displays the actions to be performed by Terraform.

    When prompted, enter yes to accept the plan and proceed with Vault configuration.

    Plan: 17 to add, 0 to change, 0 to destroy.
    
    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.
    
      Enter a value: yes
    

Verify the configuration

To validate the Terraform configured environment, make sure that VAULT_ADDR and VAULT_TOKEN environment variables are set.

Verify namespaces

Verify that nested namespaces are created as follow:

admin
 ├── education
 │   └── training
 │       └── boundary
 └── test

1. List namespaces under admin/.

   $ VAULT_NAMESPACE=admin vault namespace list
   
   Keys
   ----
   education/
   test/
   

   This verifies that admin/education and admin/test namespaces exist.

2. List namespaces under admin/education.

   $ VAULT_NAMESPACE=admin/education vault namespace list
   
   Keys
   ----
   training/
   

3. List namespaces under admin/education/training.

   $ VAULT_NAMESPACE=admin/education/training vault namespace list
   
   Keys
   ----
   boundary/
   

Verify the "admin/test" namespace settings

In the admin/test namespace, verify that the following resources exist:

• userpass auth method is enabled
• tester and admins policies exist
• student user with tester policy exists
• kv-v2 secrets engine enabled at secret

1. List auth methods in the admin/test namespace.

   $ VAULT_NAMESPACE=admin/test vault auth list
   
   Path         Type        Accessor                  Description
   ----         ----        --------                  -----------
   token/       ns_token    auth_ns_token_1e94c4e6    token based credentials
   userpass/    userpass    auth_userpass_722cf642    n/a
   

2. Verify that tester and admins policies exists.

   $ VAULT_NAMESPACE=admin/test vault policy list
   admins
   default
   tester
   

   Optionally, read the tester policy.

   $ VAULT_NAMESPACE=admin/test vault policy read tester
   
   ## Grant 'create', 'read' and 'update' permission to paths prefixed by 'secret/data/test/'
   path "secret/data/test/*" {
     capabilities = [ "create", "read", "update" ]
   }
   
   ## Manage namespaces
   path "sys/namespaces/*" {
      capabilities = [ "create", "read", "update", "delete", "list" ]
   }
   

3. Verify that student user exists.

   $ VAULT_NAMESPACE=admin/test vault read auth/userpass/users/student
   
   Key                        Value
   ---                        -----
   policies                   [tester]
   token_bound_cidrs          []
   token_explicit_max_ttl     0s
   token_max_ttl              0s
   token_no_default_policy    false
   token_num_uses             0
   token_period               0s
   token_policies             [tester]
   token_ttl                  0s
   token_type                 default
   

4. Verify that kv-v2 secrets engine is enabled at the secret path.

   $ VAULT_NAMESPACE=admin/test vault secrets list
   

   Example output:

   Path          Type            Accessor                 Description
   ----          ----            --------                 -----------
   cubbyhole/    ns_cubbyhole    ns_cubbyhole_12799478    per-token private secret storage
   identity/     ns_identity     ns_identity_bcf5f9c6     identity store
   secret/       kv              kv_37921185              n/a
   sys/          ns_system       ns_system_8b5cd0af       system endpoints used for control, policy and debugging
   

Verify the "admin/education" namespace settings

In the admin/education namespace, verify that the following resources exist:

• eaas-client and admins policies exist
• kv-v2 secrets engine is enabled at kv-v2
• transit secrets engine is enabled
  • payment key is created

1. List policies.

   $ VAULT_NAMESPACE=admin/education vault policy list
   admins
   default
   eaas-client
   

   Optionally, read the eaas-client policy.

   $ VAULT_NAMESPACE=admin/education vault policy read eaas-client
   
   ## Permits CRUD operation on kv-v2
   path "kv-v2/data/*" {
     capabilities = ["create", "read", "update", "delete", "list"]
   }
   
   ## Encrypt data with 'payment' key
   path "transit/encrypt/payment" {
     capabilities = ["update"]
   }
   
   ## Decrypt data with 'payment' key
   path "transit/decrypt/payment" {
     capabilities = ["update"]
   }
   
   ## Read and list keys under transit secrets engine
   path "transit/*" {
     capabilities = ["read", "list"]
   }
   

2. Verify that kv-v2 secrets engine is enables at kv-v2, and transit secrets engine is enabled at transit.

   $ VAULT_NAMESPACE=admin/education vault secrets list
   

   Example output:

   Path          Type            Accessor                 Description
   ----          ----            --------                 -----------
   cubbyhole/    ns_cubbyhole    ns_cubbyhole_d151ea25    per-token private secret storage
   identity/     ns_identity     ns_identity_55b10e36     identity store
   kv-v2/        kv              kv_0c232f1f              n/a
   sys/          ns_system       ns_system_f5e4482d       system endpoints used for control, policy and debugging
   transit/      transit         transit_6c1a7992         n/a
   

3. Verify that payment key exists.

   $ VAULT_NAMESPACE=admin/education vault list transit/keys
   
   Keys
   ----
   payment
   

Verify policies

You already verified that the admins policy exists in the admin/education and admin/test namespaces. Now, verify that the policy exists in the admin, admin/education/training and admin/education/training/boundary namespaces.

1. Check the admin namespace.

   $ VAULT_NAMESPACE=admin vault policy list
   admins
   default
   hcp-root
   

2. Check the admin/education/training namespace.

   $ VAULT_NAMESPACE=admin/education/training vault policy list
   admins
   default
   

3. Check the admin/education/training/boundary namespace.

   $ VAULT_NAMESPACE=admin/education/training/boundary vault policy list
   admins
   default
   

Scale an HCP Vault Dedicated cluster up or down

During the Deploy HCP Vault Dedicated with Terraform tutorial you created an Vault Dedicated cluster using the Development tier by setting the tier variable in the variables.tf file to dev.

Now that the Vault Dedicated cluster has been created, you can scale the Vault Dedicated cluster up or down to meet organizational needs by changing the tier variable and re-running terraform apply.

1. Open the variables.tf file.

2. Locate the tier variable code block

     variable "tier" {
       description = "Tier of the Vault Dedicated cluster. Valid options for tiers."
       type        = string
       default     = "dev"
     }
   

3. Change default from "dev" to "standard-small" and save the changes.

     variable "tier" {
       description = "Tier of the Vault Dedicated cluster. Valid options for tiers."
       type        = string
       default     = "starter_small"
     }
   

   Note

   Valid tier sizes include: dev, starter_small, standard_small, standard_medium, standard_large.

4. Execute the apply command to configure Vault.

   $ terraform apply
   

   This displays the actions to be performed by Terraform.

   When prompted, enter yes to accept the plan and proceed with Vault configuration.

   Plan: 0 to add, 1 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes
   

5. The Vault Dedicated cluster will be scaled up from the development tier to the starter_small tier.

Clean up

When you are done exploring, you can undo the configuration made by Terraform.

The admin token generated by the HCP Portal is valid for 6 hours. If it is expired, you need to generate a new one and overwrite the VAULT_TOKEN value to proceed.

1. Destroy the Vault resources created by Terraform.

   $ terraform apply -destroy -auto-approve
   
   ...snip...
   
   Destroy complete! Resources: 17 destroyed.
   

2. Remove the terraform state files.

   $ rm *.tfstate.*
   

3. Unset the VAULT_TOKEN environment variable.

   $ unset VAULT_TOKEN
   

4. Unset the VAULT_ADDR environment variable.

   $ unset VAULT_ADDR
   

Help and reference

• Terraform Vault Provider documentation page
• Terraform Provider GitHub repository
• Learn Terraform