HashiTalks 2025 Learn about unique use cases, homelab setups, and best practices at scale at our 24-hour virtual knowledge sharing event. Register
Vault
Vault Home

Vault with AWS KMS external key store (XKS) via PKCS#11 and XKS proxy

Enterprise

Appropriate Vault Enterprise  license required

Note: AWS xks-proxy is used in this document as a sample implementation.

Vault's KMIP Secrets Engine can be used as an external key store for the AWS KMS External Key Store (XKS) protocol using the AWS xks-proxy along with the Vault PKCS#11 Provider.

Overview

This is tested as working with Vault 1.11.0 Enterprise (and later) with Advanced Data Protection (KMIP support).

Prerequisites:

  • A server capable of running XKS Proxy on port 443, which is exposed to the Internet or a VPC endpoint. This can be the same as the Vault server.
  • A valid DNS entry with a valid TLS certificate for XKS Proxy.
  • libvault-pkcs11.so downloaded from releases.hashicorp.com for your platform and available on the XKS Proxy server.
  • Vault Enterprise with the KMIP Secrets Engine available and with TCP port 5696 accessible to where XKS Proxy will be running.

There are 3 parts to this setup:

  1. Vault KMIP Secrets Engine standard setup. (There is nothing specific to XKS in this setup.)
  2. Vault PKCS#11 setup to tell the PKCS#11 provider (libvault-pkcs11.so) how to talk to the Vault KMIP Secrets Engine. (There is nothing specific to XKS in this setup.)
  3. XKS Proxy setup.

Important: XKS has a strict 250 ms latency requirement. In order to serve requests with this latency, we recommend hosting Vault and the XKS proxy as close as possible to the desired KMS region.

Vault setup

On the Vault server, we need to setup the KMIP Secrets Engine:

  1. Start the KMIP Secrets Engine and listener:

    vault secrets enable kmip
vault write kmip/config listen_addrs=0.0.0.0:5696

  2. Create a KMIP scope to contain the AES keys that will be accessible. The KMIP scope is essentially an isolated namespace. Here is an example creating one called my-service (which will be used throughout this document).

    vault write -f kmip/scope/my-service

  3. Create a KMIP role that has access to the scope:

    vault write kmip/scope/my-service/role/admin operation_all=true

  4. Create TLS credentials (a certificate, key, and CA bundle) for the KMIP role:

    Note: This command will output the credentials in plaintext.

    vault write -f -format=json kmip/scope/my-service/role/admin/credential/generate | tee kmip.json

    The response from the credential/generate endpoint is JSON. The .data.certificate entry contains a bundle of the TLS client key and certificate we will use to connect to KMIP with from xks-proxy. The .data.ca_chain[] entries contain the CA bundle to verify the KMIP server's certificate. Save these to, e.g., cert.pem and ca.pem:

    jq --raw-output --exit-status '.data.ca_chain[]' kmip.json > ca.pem
jq --raw-output --exit-status '.data.certificate' kmip.json > cert.pem

XKS proxy setup

The rest of the steps take place on the XKS Proxy server.

For this example, We will use an HTTPS proxy service like ngrok to forward connections to the XKS proxy. This helps to quickly setup a valid domain and TLS endpoint for testing.

  1. Start ngrok:

    $ ngrok http 8000

    This will output a domain that can be used to configure KMS later, such as https://example.ngrok.io.

  2. Copy the libvault-pkcs11.so binary to the server, such as /usr/local/lib (should be same as in the TOML config file below), and chmod it so that it is executable.

  3. Copy the TLS certificate bundle (e.g., /etc/kmip/cert.pem) and CA bundle (e.g., /etc/kmip/ca.pem) to the xks-proxy server (doesn't matter where, as long as the xks-proxy process has access to it) from the Vault setup.

  4. Create a configuration/settings_vault.toml file for the XKS to Vault PKCS#11 configuration, and set the XKS_PROXY_SETTINGS_TOML environment variable to point to the file location.

    The important settings to change:

    • [[external_key_stores]]:
      • change URI path prefix to anything you like
      • choose random access ID
      • choose random secret key
      • set which key labels are accessible to XKS (xks_key_id_set)
    • [pkcs11]: set the PKCS11_HSM_MODULE to the location of the libvault-pkcs11.so (or .dylib) file downloaded from releases.hashicorp.com.
    [server]
ip = "0.0.0.0"
port = 8000
region = "us-east-2"
service = "kms-xks-proxy"

[server.tcp_keepalive]
tcp_keepalive_secs = 60
tcp_keepalive_retries = 3
tcp_keepalive_interval_secs = 1

[tracing]
is_stdout_writer_enabled = true
is_file_writer_enabled = true
level = "DEBUG"
directory = "/var/local/xks-proxy/logs"
file_prefix = "xks-proxy.log"
rotation_kind = "HOURLY"

[security]
is_sigv4_auth_enabled = true
is_tls_enabled = true
is_mtls_enabled = false

[tls]
tls_cert_pem = "tls/server_cert.pem"
tls_key_pem = "tls/server_key.pem"
mtls_client_ca_pem = "tls/client_ca.pem"
mtls_client_dns_name = "us-east-2.alpha.cks.kms.aws.internal.amazonaws.com"

[[external_key_stores]]
uri_path_prefix = "/xyz"
sigv4_access_key_id = "AKIA4GBY3I6JCE5M2HPM"
sigv4_secret_access_key = "1234567890123456789012345678901234567890123="
xks_key_id_set = ["abc123"]

[pkcs11]
session_pool_max_size = 30
session_pool_timeout_milli = 0
session_eager_close = false
user_pin = ""
PKCS11_HSM_MODULE = "/usr/local/lib/libvault-pkcs11.so"
context_read_timeout_milli = 100

[limits]
max_plaintext_in_base64 = 8192
max_aad_in_base64 = 16384

[hsm_capabilities]
can_generate_iv = false
is_zero_iv_required = false

Note: vault-pkcs11-provider versions of 0.1.0–0.1.2 require the last two lines to be changed to can_generate_iv = true and is_zero_iv_required = true.

  1. Create a file, /etc/vault-pkcs11.hcl with the following contents:

    slot {
  server = "VAULT_ADDRESS:5696"
  tls_cert_path = "/etc/kmip/cert.pem"
  ca_path = "/etc/kmip/ca.pem"
  scope = "my-service"
}

    This file is used by libvault-pkcs11.so to know how to find and communicate with the KMIP server. See the Vault docs for all available parameters and their usage.

  2. If you want to view the Vault logs (helpful when trying to find error messages), you can specify the VAULT_LOG_FILE (default is stdout) and VAULT_LOG_LEVEL (default is INFO). We'd recommend setting VAULT_LOG_FILE to something like /tmp/vault.log or /var/log/vault.log. Other useful log levels are WARN (quieter) and TRACE (very verbose, could possibly contain sensitive information, like raw network packets).

  3. Create an AES-256 key in KMIP, for example, using pkcs11-tool (usually installed with the OpenSC package). See the Vault docs for the full setup.

    VAULT_LOG_FILE=/dev/null pkcs11-tool --module ./libvault-pkcs11.so --keygen -a abc123 --key-type AES:32 \
    --extractable --allow-sw
Key generated:
Secret Key Object; AES length 32
VALUE:
label:      abc123
Usage:      encrypt, decrypt, wrap, unwrap
Access:     none

Enable XKS in the AWS CLI

  1. Create the KMS custom key store with the appropriate parameters to point to your XKS proxy (in this example, through ngrok).

    $ aws kms create-custom-key-store \
    --custom-key-store-name myVaultKeyStore \
    --custom-key-store-type EXTERNAL_KEY_STORE \
    --xks-proxy-uri-endpoint https://example.ngrok.io \
    --xks-proxy-uri-path /xyz/kms/xks/v1 \
    --xks-proxy-authentication-credential AccessKeyId=AKIA4GBY3I6JCE5M2HPM,RawSecretAccessKey=1234567890123456789012345678901234567890123= \
    --xks-proxy-connectivity PUBLIC_ENDPOINT

{
    "CustomKeyStoreId": "cks-d7a55fe93d63191d6"
}

  2. Tell KMS to connect to the key store.

    $ aws kms connect-custom-key-store --custom-key-store-id cks-d7a55fe93d63191d6

  3. Wait for the ConnectionState of your custom key store to be CONNECTED. This can take a few minutes.

    $ aws kms describe-custom-key-stores --custom-key-store-id cks-d7a55fe93d63191d6

  4. Create a KMS key associated with the XKS key ID (abc123 in this example):

    $ aws kms create-key --custom-key-store-id cks-d7a55fe93d63191d6 \
    --xks-key-id abc123 --origin EXTERNAL_KEY_STORE
{
    "KeyMetadata": {
        "AWSAccountId": "111111111111",
        "KeyId": "a93f205a-2a37-4338-aa64-92b4a4b0b67d",
        "Arn": "arn:aws:kms:us-east-2:111111111111:key/a93f205a-2a37-4338-aa64-92b4a4b0b67d",
        "CreationDate": "2022-12-22T11:03:23.695000-08:00",
        "Enabled": true,
        "Description": "",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "EXTERNAL_KEY_STORE",
        "CustomKeyStoreId": "cks-16460f66b34705025",
        "KeyManager": "CUSTOMER",
        "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
        "KeySpec": "SYMMETRIC_DEFAULT",
        "EncryptionAlgorithms": [
            "SYMMETRIC_DEFAULT"
        ],
        "MultiRegion": false,
        "XksKeyConfiguration": {
            "Id": "abc123"
        }
    }
}

  5. Encrypt some data with this key:

    $ aws kms encrypt --key-id a93f205a-2a37-4338-aa64-92b4a4b0b67d --plaintext YWJjMTIzCg==
{
    "CiphertextBlob": "somerandomciphertextblob=",
    "KeyId": "arn:aws:kms:us-east-2:111111111111:key/a93f205a-2a37-4338-aa64-92b4a4b0b67d",
    "EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
}

  6. Decypt the resulting ciphertext:

    $ aws kms decrypt --ciphertext-blob somerandomciphertextblob=
{
    "KeyId": "arn:aws:kms:us-east-2:111111111111:key/a93f205a-2a37-4338-aa64-92b4a4b0b67d",
    "Plaintext": "YWJjMTIzCg==",
    "EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
}

  7. Optionally, clean up your key and key store with:

    $ aws kms disable-key --key-id a93f205a-2a37-4338-aa64-92b4a4b0b67d
$ aws kms disconnect-custom-key-store --custom-key-store-id cks-16460f66b34705025
$ aws kms delete-custom-key-store --custom-key-store-id cks-16460f66b34705025

    (The aws kms delete-custom-key-store command will not succeed until all keys in the key store have been disabled and deleted.)

Edit this page on GitHub