Recommended pattern for stateless Vault for Transit auto unseal

Objective

This document aims to provide a framework for creating a usable solution for auto unseal using HashiCorp Vault when HSM or cloud-based KMS auto unseal mechanism is not available for your environment, such as in an internal Data Center deployment.

Prerequisites

We assume a primary Vault cluster configuration aligned with our Reference Architecture guide. Additionally, we assume familiarity with Vault authentication mechanisms.

Glossary

• Region: A networking environment that is private, low latency, and high bandwidth. This excludes communication that would traverse the public internet and typically would be a single physical data center or a cloud region. A region typically comprises one or more availability zones.
• Cluster: A complete deployment unit for operating Vault in a highly-available manner. Consists of one active Vault node and any number of standby and/or performance standby Vault nodes. All nodes in a single cluster share the same storage backend for persisting data.
• Primary Cluster: Vault Enterprise replication modes use a leader-follower pattern. We refer to the leader cluster as the primary cluster. In a Vault replication relationship, the primary cluster is responsible for writing all shared data to the configured storage backend.
• Secondary Cluster: Vault Enterprise replication modes use a leader-follower pattern. We refer to the follower cluster as the secondary cluster. Data is streamed from the primary cluster to all configured secondary clusters.

Vault transit overview

The transit secrets engine handles cryptographic functions on data in transit. Vault doesn't store the data sent to the secrets engine. It is also referred to as "cryptography as a service" or "encryption as a service." The transit secrets engine can also sign and verify data, generate hashes and HMACs of data, and act as a random bytes source.

The primary use case for the transit secrets engine is to encrypt data from applications while still storing that encrypted data in some primary data store. This relieves the burden of proper encryption/decryption from application developers and pushes the responsibility onto Vault's operators.

Key derivation is supported, which allows you to use the same key for multiple purposes by deriving a new key based on a user-supplied context value. In this mode, transit secrets engine can support convergent encryption, allowing the same input values to produce the same ciphertext.

Datakey generation allows processes to request a high-entropy key of a given bit length be returned to them, encrypted with the named key. Usually, this also returns the key in plaintext to allow for immediate use, but you can disable this for auditing requirements.

Vault transit auto unseal overview

Combining the above concepts, Vault can use an external Vault cluster’s mounted transit secrets engine to be the trusted system for decrypting unseal material, allowing for the automation of unseal operations. This operation assumes the sealed Vault server knows the location of its unsealing Vault, as well as a valid token with appropriate permissions to encrypt and decrypt against a known transit key.

Transit unseal configuration

The transit seal configures Vault to use Vault's transit secrets engine as the auto unseal mechanism. The Transit seal is activated by one of the following: the presence of a seal "transit" block in Vault's configuration file or the environment variable VAULT_SEAL_TYPE set to transit.

seal "transit" {
  address            = "https://vault:8200"
  token              = "s.Qf1s5zigZ4OX6akYjQXJC1jY"
  disable_renewal    = "false"

  // Key configuration
  key_name           = "transit_key_name"
  mount_path         = "transit/"
  namespace          = "ns1/"

  // TLS Configuration
  tls_ca_cert        = "/etc/vault/ca_cert.pem"
  tls_client_cert    = "/etc/vault/client_cert.pem"
  tls_client_key     = "/etc/vault/ca_cert.pem"
  tls_server_name    = "vault"
  tls_skip_verify    = "false"
}

Transit parameters

The following parameters apply to the seal stanza in the Vault configuration file.

Recommended unsealing Vault configuration

It's a good idea to provide unsealing facilities to your Vault clusters by standing up a minimally configured Vault cluster solely to hold the Transit Key for the unseal operation. This Vault is not meant to serve general client requests and has significantly lower resilience requirements. The desire is often for a small “Unseal as a Service” Vault with minimal overhead and complexity.

However, keep in mind that this unseal Vault will become an entity in the chain of protection of your primary Vault cluster’s secret data, and as such, should still be considered to contain sensitive material.

Typically, we have seen the unsealing Vault deployed as a small virtual machine or container. Often it is run as a single node without a running standby. For simplicity, use a non-HA supporting backend is also common, such as a file backend.

Required policy

The primary Vault cluster will require a Vault token valid on the unsealing Vault to unseal itself. This token can be fetched through the Vault Agent, or a scripted process in systemd, for example. Acquire this token through authentication with trusted material, such as leveraging LDAP or an AppRole authentication scheme. It is recommended that you populate the Vault token as an environment variable instead of written to the Vault configuration file.

The returned Vault token must be associated with a Vault Policy that gives appropriate permissions to the encrypt and decrypt endpoints of the Transit key in use. Below is an example policy.

path "<mount path>/encrypt/<key_name>" {
  capabilities = ["update"]
}

path "<mount path>/decrypt/<key_name>" {
  capabilities = ["update"]
}

Deployment considerations

There are a few challenges to be considered in more complex Vault topologies. Some of these concerns are universally applicable to the pattern of using an unsealing Vault, and some are more specific to Vault topologies that leverage replication between multiple data centers.

Unsealing the unseal Vault

The unsealing Vault itself still requires unsealing when being started. Effectively, this pattern merely moves the burden of non-automated unseal from your production-serving Vault cluster to the supporting service unsealing Vault. Managing the manual effort on a service will subject that service to a less strenuous load.

Unseal Vault address

As the unseal operation requires a target Vault server address as an environment variable or value in the configuration, a reliable reference to the unsealing Vault’s network location is needed. You can solve this issue most elegantly by the use of a service discovery system, such as Consul. If such a tool is not available, leverage an abstracted reference to the services location, such as a managed DNS entry, load balancer VIP, or Anycast IP address, based on your capabilities and leveraged patterns in your environment.

Replication concerns

When working in a multi-site deployment, with Vault replication configured between primary Vault clusters, additional concerns may arise for unsealing behavior. Each cluster in the replication configuration requires a known unsealing Vault. While it is possible to use a single unsealing Vault for all replicated clusters, network connectivity between all clusters and a single site would require network connectivity. We recommend providing an unsealing Vault in each data center.

Key rotation

When rotating the unsealing key on the unsealing Vault servers, run the rotation operation on each of the Vault primaries in each data center. You can use the same configuration on all unsealing Vaults in some scenarios, with its storage backend point-in-time replicated out to each data center. If you choose this option, it is recommended that you use a unique key for each primary cluster, so that rotation operations are consistent, and run each rotation against the same unsealing Vault. Otherwise, you must carry out replication actions between each rotation.