Sync secrets with Vault Enterprise

10min
|
Enterprise
Vault
Video
Interactive

Vault 1.16 includes a new feature for HashiCorp Vault Enterprise - Secrets sync. This allows you to automatically sync secrets from Vault Enterprise to a variety of third party platforms including AWS, Azure, GCP, GitHub, and Vercel.

This tutorial instructs the learner to create a static K/V secret in Vault Enterprise. Then you set up a destination and association to sync that secret between HashiCorp Vault Enterprise and AWS Secrets Manager.

Challenge

There are two challenges are addressed by Enterprise Secrets Sync.

First, there are some systems that cannot communicate with Vault. Cloud providers naturally offer strong support with their own secrets management solutions. For example, AWS RDS Proxy can only communicate with AWS Secrets Manager. This contributes to secrets sprawl, and being able to manage secrets in Vault Enterprise but allow access through platform specific tools is invaluable.

Second, organizations are interested in leveraging the power of secrets management with Vault but access them in a platform native way. Though there are many resources available, developers are required to learn new technologies to leverage the power of Vault Enterprise. Secrets Sync allows developers to focus on their preexisting platform knowledge and reduce effort an organization to adopt Vault. By focusing on cloud provider tools, they build on preexisting skills.

Solution

secrets sync example

Vault Enterprise Secrets Sync will automatically sync secrets between Vault and AWS Secrets Manager so that your developers do not need to learn how to consume secrets from Vault. Operators can still leverage the power of Vault Enterprise for secrets lifecycle management, but developers can consume the secrets through platform native tools and technologies.

associations and destinations Sync destinations will represent a location where secrets can be synced with. Currently there are five types of destinations: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, GitHub repository secret and Vercel Project environment variables. One secret can be associated with multiple destinations. For example, a Jira token stored as a Vault secret used in CI workflows could be synced into multiple GitHub repositories, each repository being represented by a distinct Vault sync destination.

Secret associations represent a link between an existing Vault KV secret and a sync destination. Associations are granular and each secrets has a association. Many associations can share a destination.

Personas

This tutorial is intended for a Vault administrator using Vault Enterprise for secrets lifecycle management but their organization requires that the secrets to be available through AWS Secrets Manager.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

To complete this tutorial you will need the following:

Vault Enterprise version 1.16 or later with premium license.
Vault CLI
Terraform CLI
Amazon Web Services test account. This tutorial requires the creation of new cloud resources and but should not incur costs associated with the deployment and management of these resources. Your IAM user will need permission to create IAM User, and to create access keys for the user.
- Basic understanding of the AWS user interface, and the use of AWS Secrets Manager.
- If creating an IAM user with Terraform, your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are required.

Lab set up

To set up this lab you will need to clone an Github repository, create an IAM user with the proper policy, and start an local instance of Enterprise Vault.

Create AWS IAM User

In order to set up secrets sync Vault needs programmatic access to create, read, delete, update secrets in AWS Secrets Manager. In this section you will create an IAM User with least privileged access needed to accomplish this. You can create the IAM user manually or you can run the Terraform config included in the GitHub repository.

Create a IAM policy named vault_secrets_sync_policy with the following privileges.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Sid": "SecretsManagerAccess",
            "Effect": "Allow",
            "Action": [
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetSecretValue",
               "secretsmanager:CreateSecret",
               "secretsmanager:PutSecretValue",
               "secretsmanager:UpdateSecret",
               "secretsmanager:UpdateSecretVersionStage",
               "secretsmanager:DeleteSecret",
               "secretsmanager:RestoreSecret",
               "secretsmanager:TagResource",
               "secretsmanager:UntagResource"
            ],
            "Resource": "*"
      }
   ]
}

Create a IAM user named vault_secrets_sync_user and attach the vault_secrets_sync_policy.
Create AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the IAM user. See Managing access keys (console) for instructions.
Open a new terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY environment variables to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION below.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Clone the repository at learn-vault-enterprise-secrets-sync.

$ git clone https://github.com/hashicorp-education/learn-vault-enterprise-secrets-sync.git

Move into the terraform subdirectory of folder just created.
```
$ cd learn-vault-enterprise-secrets-sync/terraform
```
Working directory
This tutorial assumes that the remainder of commands in the Terraform section are executed in this directory.
Export your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables. This key pair should have permission to create the AWS IAM resources in the provided Terraform configuration.
If you are not running on us-east-1 please update the AWS_REGION below.
```
$ export AWS_ACCESS_KEY_ID=<<YOUR_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<YOUR_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Initialize the configuration.

$ terraform init
...
Terraform has been successfully initialized!
...

Look at the plan.

$ terraform plan
...
Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ access_key = (known after apply)
+ secret_key = (sensitive value)
...

Apply the configuration.

$ terraform apply -auto-approve

...

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

access_key = "AKIA3SCJYFFCO3FDO4P3"
secret_key = <sensitive>

In the terraform.tfstate file to locate the secret access key.

$ cat terraform.tfstate | head -n 15
{
"version": 4,
"terraform_version": "1.5.4",
"serial": 4,
"lineage": "76f54aef-b09b-647a-ab25-803ecb8f5a1f",
"outputs": {
   "access_key": {
      "value": "AKIA4V3NG3RZO3FDO4P3",
      "type": "string"
   },
   "secret_key": {
      "value": "tXMencyCLQpz1sRowsaMFGEZbsibIDJli9/Qy/E",
      "type": "string",
      "sensitive": true
   }
},

Near the top of the block just examined will find outputs block with a access_key and secret_key fields. Copy the value for each.
In the same terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION below.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Set up Vault Enterprise

Open a new terminal.
Set your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.
```
$ export VAULT_LICENSE=02MV4UU43BK5....
```
Start a Vault development server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault development server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault development server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Return to the other terminal session.
Export an environment variable for the Vault CLI to the address of the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the Vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

Check the status of Vault.

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.16.0+ent
Build Date      2023-09-08T17:10:08Z
Storage Type    inmem
Cluster Name    vault-cluster-42e92bf5
Cluster ID      fXXaMe47-nb77-2be6-9426-a7986s376t25
HA Enabled      false

The Vault server is ready.

Enterprise secrets sync

Create a KV secret

Create the KV secrets engine.

$ vault secrets enable -path=kv1 kv-v2
Success! Enabled the kv-v2 secrets engine at: kv1/

Create the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="123"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T17:56:26.700919Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Configure destination and association

Activate the secrets sync feature from the root namespace.

$ vault write -f sys/activation-flags/secrets-sync/activate

Configure the destination.

$ vault write sys/sync/destinations/aws-sm/aws-sm1 \
   access_key_id="${VAULT_USER_ACCESS_KEY_ID}" \
   secret_access_key="${VAULT_USER_SECRET_ACCESS_KEY}" \
   secret_name_template="{{ .MountAccessor }}_{{ .SecretBaseName }}"
   region="${AWS_REGION}"

The output should resemble the following:

Key                   Value
---                   -----
connection_details    map[access_key_id:***** region:us-east-1 secret_access_key:*****]
name                  aws-sm1
options               map[custom_tags:map[] granularity_level:secret-path secret_name_template:{{ .MountAccessor }}_{{ .SecretBaseName }}]
type                  aws-sm

By default, the name of synced secrets follows this format: vault/<accessor>/<secret-path>, but each destination allows you to customize the name through the secret_name_template field. Refer to the Name Template documentation for details.

Set up the association.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/set \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[kv_c6c5fb8c/database/dev:map[accessor:kv_c6c5fb8c secret_name:database/dev]]
store_name            aws-sm1
store_type            aws-sm

Log into your AWS Account and take a navigate to the AWS Secrets Manager.
Choose the link ending with /database/dev.
In AWS Secrets Manager choose Retrieve secret value in the Secret value section.
These are the same values as the kv secret in kv1/database/dev.

Modifications to the secret

Changes to the secret in Vault Enterprise are also reflected in AWS Secrets Manager.

Update the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="updated"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-12T18:36:42.241359Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Open up your AWS account and reload the secret in AWS Secrets Manager.

Use a a patch command to add an additional field.

$ vault kv patch -mount=kv1 database/dev last_key=new_one
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T18:00:49.720538Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3

Remove associations (optional)

If you no longer want the secret to sync with AWS Secrets Manager, you can remove the association.

Remove the association between the secret and AWS Secrets Manager.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/remove \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[]
store_name            aws-sm1
store_type            aws-sm

Return to your browser windows with AWS Secrets Manager and refresh the page. It should resemble the following:

Clean up

If you created the IAM user manually, delete the IAM user.
If you created the IAM User with Terraform, switch back to the original AWS credentials AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY used with Terraform to create the IAM User.

Destroy the resources Terraform created.

$ terraform destroy -auto-approve

...

Destroy complete! Resources: 3 destroyed.

Unset the environment variables used in this lab.

$ unset VAULT_USER_ACCESS_KEY \
VAULT_USER_SECRET_ACCESS_KEY \
VAULT_ADDR \
VAULT_TOKEN \

Stop the Vault server process.
```
$ pgrep -f vault | xargs kill
```
If you were doing the terraform version of this tutorial, go back to where the learn-vault-enterprise-secrets-sync repository was cloned.
```
$ cd ../..
```

Delete the learn-vault-pki-engine directory.

$ rm -rf learn-vault-enterprise-secrets-sync

Summary

You learned how to set up Vault Enterprise secrets sync, which consists of destinations and associations. Destinations are platforms that a secret will be synced from Vault to. Associations create a link between a secret and sync destination. When the secret is changed in Vault it will change in the target platform service.

This tutorial guided you to set up a link between a secret in Vault Enterprise and AWS Secrets Manager, and demonstrated that changes to the secret in Vault are reflected in AWS Secrets Manager.

Note that this tutorial does not cover performance replication with secrets sync in mind. This is because most sync operations are mainly performed by only the active node within a Vault cluster. Write operations, which are what most sync operations will be, does not benefit from scaling.

References

Certificate Issuance External Policy Service

Federated workload identities