HashiConf 2024 Now streaming live from Boston! Attend for free
Vault
Ops fundamentals

Emergency break-glass features

  • 23min
  • |
  • Enterprise
  • Vault

Emergency situations or unexpected behavior can arise when operating any software, with Vault being no exception.

Learn how to use critical Vault features with your processes and procedures for swiftly responding to incidents.

Challenge

Operators and responders require features and functionality which effectively support break-glass procedures used in incident response.

When incidents such as credential leakage, intrusion, or denial of service (DOS) attacks occur, timely mitigation is of the essence when supporting business operational goals.

Solution

Learn about and use these two important Vault features which are designed to aid operators in effectively responding to incidents.

  • Seal: In a typical operation, Vault runs in an unsealed state. However, an operator with sufficient privileges can seal the running Vault at any time. When sealed, the Vault server discards its in-memory key for unlocking data, and is therefore physically blocked from responding to all requests with the exception of status checks and unsealing.

  • API Lock: (new in v1.9.0) A namespace API lock feature was added to provide more granular access lockout. If an operator does not require that Vault be entirely sealed, they can choose instead to lock only the API on a per-namespace level, and prevent Vault from responding to all but status and unlocking requests.

Prerequisites

To perform all the steps in the hands on scenario, you need:

  • Vault Enterprise v1.9 or later binary installed in your system path.

  • jq is used to pretty print JSON output examples.

Policy requirements

Note

For the purpose of this tutorial, you can use the root token to work with Vault. However, it is recommended that root tokens are used for just enough initial setup or in emergencies. As a best practice, use tokens with an appropriate set of policies based on your role in the organization. Unless you are running Vault locally in development mode (-dev), it is recommended that you authenticate with Vault and acquire a token with an appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

If you are connecting to a non-development mode Vault server, your token must have a policy with the following permissions:

# Seal Vault
path "sys/seal" {
  capabilities = [ "create", "update", "sudo" ]
}

# Unseal Vault
path "sys/unseal" {
  capabilities = [ "create", "update", "sudo" ]
}

# Lock and unlock API per namespace
path "/sys/namespaces/api-lock/+/*" {
  capabilities = [ "create", "update", "sudo" ]
}

If you are not familiar with policies, complete the policies tutorial.

Personas

The end-to-end scenario described in this tutorial involves two personas:

  • operator with privileged capabilities for sealing and unsealing Vault, along with locking and unlocking API endpoints.

  • driver uses the username and password auth method enabled within the drivers namespace to authenticate with Vault.

Scenario introduction

As shown at left, when Vault is sealed, all users are affected and unable to use it.

Contrast this with locking the API at the namespace level as shown on the right. Although the drivers namespace within the teams namespace is locked, users in the teams and root namespaces are unaffected by the lock.

Scenario diagram

You will use terminal sessions to operate a single Vault server with Integrated Storage (Raft), and follow a series of hands-on scenarios to learn how to seal and unseal Vault from the Vault CLI, HTTP API, or web UI.

You will then learn how to lock the Vault API on a per-namespace basis from the Vault CLI or HTTP API with a username and password auth method example.

You can also clean up the scenario environment by using the provided example commands when you are finished learning.

Establish a scenario environment

Make a temporary directory to hold the files created for this scenario, and assign its path to the environment variable LEARN_VAULT.

$ mkdir -p /tmp/learn-vault/data && export LEARN_VAULT=/tmp/learn-vault

Create Vault server configuration

Create a minimal configuration for a single Vault server that uses Raft storage.

$ cat > $LEARN_VAULT/vault-server.hcl << EOF
api_addr                = "http://127.0.0.1:8200"
cluster_addr            = "http://127.0.0.1:8201"
cluster_name            = "learn-vault"
default_lease_ttl       = "10h"
disable_mlock           = true
max_lease_ttl           = "10h"
ui                      = true

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_disable   = "true"
}

backend "raft" {
  path    = "/tmp/learn-vault/data"
  node_id = "learn-vault-1"
}
EOF

Start a single Vault server

Open a terminal session and start a Vault server using the configuration that you just created.

$ vault server -config $LEARN_VAULT/vault-server.hcl

The server starts uninitialized and sealed.

Initialize, unseal & authenticate

In another terminal session, export the VAULT_ADDR environment variable to address the Vault server and export the LEARN_VAULT environment variable to define the scenario directory.

$ export VAULT_ADDR=http://127.0.0.1:8200 LEARN_VAULT=/tmp/learn-vault

For the purpose of simplicity in this tutorial, initialize Vault with 1 key share and a key threshold of 1 and write the output to the file .vault-init in the project directory.

$ vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    | head -n3 \
    | cat > $LEARN_VAULT/.vault-init

Successful execution of this command should produce no output.

Export the unseal key value as the environment variable UNSEAL_KEY.

$ export UNSEAL_KEY=$(grep 'Unseal Key 1'  $LEARN_VAULT/.vault-init | awk '{print $NF}')

Unseal Vault with the Unseal Key 1 value from the .vault-init file.

$ vault operator unseal $UNSEAL_KEY

Successful output from unsealing Vault should resemble this example:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.9.0+ent
Storage Type            raft
Cluster Name            learn-vault
Cluster ID              fb4c6314-f2df-2cbc-aa21-9e794f44003a
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    55
Raft Applied Index      55

Export the initial root token value to the environment variable ROOT_TOKEN.

$ export ROOT_TOKEN=$(grep 'Initial Root Token' $LEARN_VAULT/.vault-init | awk '{print $NF}')

login with vault login by passing the Initial Root Token value from the .vault-init file.

$ vault login -no-print $ROOT_TOKEN

This command should produce no output when successful. If you want to confirm that the login was successful, try a token lookup and confirm that your token policies contain root.

Note

You will use a root token in this scenario for simplicity. However, in actual production environments, root tokens should be closely guarded and used only for tightly controlled purposes. Review the documentation on root tokens for more details.

$ vault token lookup | grep policies

Successful output should contain the following.

policies            [root]

You are ready for the first scenario.

Seal Vault

Vault running in an unsealed state uses the master encryption key to decrypt the encryption key kept in Vault storage that is itself used for encrypting and decrypting secrets in storage. Vault is sealed with the CLI from a terminal session in this example; the master encryption key is discarded from memory, and no further encryption or decryption from storage is possible until Vault is unsealed again.

Seal Vault with command line diagram

If an incident such as intrusion or credential leakage occurs, sealing Vault in response is the heaviest tool you can use. The key used to decrypt data will be immediately discarded from memory, and Vault can no longer decrypt anything in its storage until unsealed again.

Note

A sealed Vault is effectively no longer available for normal use. You should keep this in mind particularly when using enterprise replication, as a sealed cluster will no longer be participating in replication. A sealed primary will prevent all secondaries from replicating with it, and a sealed secondary cluster will become out of sync with its primary.

An operator using a token with sufficient capabilities as described in the Policy requirements section can seal Vault with the vault CLI, the /sys/seal HTTP API or by using, or the Web UI.

(Persona: operator)

Seal your Vault server.

$ vault operator seal
Success! Vault is sealed.

You can check Vault status to confirm.

$ vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.9.0-rc1+ent
Storage Type       inmem_transactional_ha
HA Enabled         true

The output should show that the Sealed value is true.

You can also change to the terminal where you started the Vault server to observe the operational logging. The end of the log should resemble this example.

2021-11-14T17:23:25.131-0500 [INFO]  core.cluster-listener: rpc listeners successfully shut down
2021-11-14T17:23:25.131-0500 [INFO]  core: cluster listeners successfully shut down
2021-11-14T17:23:25.131-0500 [INFO]  core: vault is sealed

Note the last line which specifically states core: vault is sealed; this is the best signal from the logs to you as an operator that Vault is now sealed.

If you attempt to login to Vault now, you'll discover an error like the one shown in this example:

$ vault login -no-print $ROOT_TOKEN
Error authenticating: error looking up token: Error making API request.

URL: GET http://127.0.0.1:8200/v1/auth/token/lookup-self
Code: 503. Errors:

* Vault is sealed

With Vault sealed, an operator must unseal it to restore normal operations.

Note

The examples in this tutorial use the Shamir's Secret Sharing based seal with an unseal key, but in typical production installations, a cloud based automatic seal is used. Keep in mind that when a Vault server using auto unseal is sealed, it will automatically unseal itself if restarted. Use caution when restarting such servers.

Use the unseal key value recorded earlier to unseal Vault in preparation for the next scenario.

(Persona: operator)

Unseal Vault.

$ vault operator unseal $UNSEAL_KEY

Change to the terminal where you started the Vault server to observe the operational logging. The end of the log should resemble this example.

2021-11-13T16:40:16.739-0500 [INFO]  core: post-unseal setup complete
2021-11-13T16:40:16.739-0500 [INFO]  core: vault is unsealed
2021-11-13T16:40:16.740-0500 [INFO]  expiration: lease restore complete

Note the line core: vault is unsealed.

Vault is now unsealed and ready for normal operations.

Lock API

Vault Enterprise edition 1.9.0 introduced a per-namespace level API lock feature that an operator can use to lock the API for any enterprise namespace.

When the API is locked for a particular namespace, it is also locked for that namespace's descendants.

In this example, the namespace teams was locked by an operator. The drivers namespace is a descendant of teams, and as a result it was also locked. This means that the userpass auth method enabled in the drivers namespace cannot be used.

When the driver user attempts authentication with the userpass method in the drivers namespace, an error about the locked status of the namespace is returned.

Lock API in namespace

Vault blocks all API endpoints from responding to clients operating in a locked namespace or a descendant of a locked namespace with the exception of endpoints for status and unlocking the API lock.

Per namespace API locking provides a more granular tool for incident response that helps the operator to focus on locking one or more namespaces instead of the sealing the entire Vault. The feature is particularly helpful in multi-tenant environments where sealing Vault is too heavy a response that affects all tenants.

If you are unfamiliar with enterprise namespaces, review the Multi-tenancy with Namespaces tutorial.

Create namespaces and enable auth method

(Persona: operator)

Before you try the API lock feature in your Vault server, create two namespaces, a teams namespace and a drivers namespace that is a descendant of teams.

Create the teams namespace.

$ vault namespace create teams
Key     Value
---     -----
id      SpQLE
path    teams/

Create the drivers namespace as a descendant of the teams namespace.

$ vault namespace create -namespace=teams drivers
Key     Value
---     -----
id      dfTq1
path    teams/drivers/

Enable a username and password auth method in the drivers namespace.

$ vault auth enable -namespace teams/drivers userpass

Create a driver user within the drivers namespace.

$ vault write -namespace teams/drivers auth/userpass/users/driver \
      password=p@ssw0rd

Test auth method

(Persona: driver)

Test authentication of the driver user with the userpass auth method in the teams namespace.

$ vault login \
    -namespace teams/drivers \
    -method=userpass \
    username=driver \
    password=p@ssw0rd

Successful output resembles this example.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.fwOqtyX5wWQVx4TIaVkL1gLP.dfTq1
token_accessor         HjEefUHXOQy9sFWvzF0JGbcF.dfTq1
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      []
policies               ["default"]
token_meta_username    driver

Lock API in drivers namespace

(Persona: operator)

After successfully testing the authentication, login with the root token again.

$ vault login -no-print $ROOT_TOKEN

Lock the API in the drivers namespace.

$ vault namespace lock teams/drivers
Key           Value
---           -----
unlock_key    GF1pw5hU3dDNeYmSRzHxya8T

The unlock key is returned; this key must be used to later unlock the API.

Test auth method again

(Persona: driver)

With the API locked for the drivers namespace, attempt to authenticate as the user driver with the userpass auth method.

$ vault login \
    -namespace teams/drivers \
    -method=userpass \
    username=driver \
    password=p@ssw0rd

A helpful error is returned indicating that the namespace is locked.

Error authenticating: Error making API request.

Namespace: teams/drivers/
URL: PUT http://127.0.0.1:8200/v1/auth/userpass/login/driver
Code: 503. Errors:

* 1 error occurred:
  * API access to this namespace has been locked by an administrator - "teams/drivers/" must be unlocked to gain access.

Unlock API in drivers namespace

(Persona: operator)

Use the unlock key previously returned to unlock the API in the drivers namespace.

$ vault namespace unlock -unlock-key GF1pw5hU3dDNeYmSRzHxya8T teams/drivers

Successful execution results in no output.

Test auth method for the last time

(Persona: driver)

Test the authentication once more to show that the unlock was successful.

With the API locked for the drivers namespace unlocked, attempt to authenticate with the userpass auth method as the user driver.

$ vault login \
    -namespace teams/drivers \
    -method=userpass \
    username=driver \
    password=p@ssw0rd

The result should be a successful login.

Cleanup

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Unset the environment variables.

$ unset ROOT_TOKEN UNSEAL_KEY VAULT_ADDR LEARN_VAULT

You can stop the Vault server by pressing Ctrl+C where the server is running. Or, execute the following command.

$ pgrep -f vault | xargs kill

Remove the scenario environment directory.

$ rm -rf /tmp/learn-vault

Summary

You learned about two important features for incident response available in Vault. You learned about sealing an unsealing Vault, a feature available in all Vault editions.

You also learned about per namespace API locking and unlocking, a feature available only in Vault Enterprise editions.

Help and Reference

Previous

Management of Vault Enterprise with Terraform

 Next

Define custom HTTP headers