Vault lambda extension

AWS Lambda lets you run code without provisioning and managing servers. The Vault Lambda Extension utilizes the AWS Lambda Extensions API to help your Lambda function read secrets from your Vault deployment. You can use the quick-start directory which has an end-to-end example if you would like to try out the extension from scratch.

Usage

To use the extension, include one of the following ARNs as a layer in your Lambda function, depending on your desired architecture.

amd64 (x86_64):

arn:aws:lambda:<your-region>:634166935893:layer:vault-lambda-extension:18

arm64:

arn:aws:lambda:<your-region>:634166935893:layer:vault-lambda-extension-arm64:6

Where region may be any of af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ca-central-1, eu-central-1, eu-north-1, eu-south-1, eu-west-1, eu-west-2, eu-west-3, me-south-1, sa-east-1, us-east-1, us-east-2, us-west-1, us-west-2.

The extension authenticates with Vault using AWS IAM auth, and all configuration is supplied via environment variables. There are two methods to read secrets, which can both be used side-by-side:

• Recommended: Make unauthenticated requests to the extension's local proxy server at http://127.0.0.1:8200, which will add an authentication header and proxy to the configured VAULT_ADDR. Responses from Vault are returned without modification.
• Configure environment variables such as VAULT_SECRET_PATH for the extension to read a secret and write it to disk.

Adding the extension to your existing lambda and Vault infrastructure

Requirements

• ARN of the role your Lambda runs as
• An instance of Vault accessible from AWS Lambda
• An authenticated vault client
• A secret in Vault that you want your Lambda to access, and a policy giving read access to it
• Your Lambda function must use one of the supported runtimes for extensions

Step 1. configure Vault

Enable the aws auth method.

$ vault auth enable aws

Configure the AWS client to use the default options.

$ vault write -force auth/aws/config/client

Create a role prefixed with the AWS environment name.

$ vault write auth/aws/role/vault-lambda-role \
    auth_type=iam \
    bound_iam_principal_arn="${YOUR_ARN}" \
    policies="${YOUR_POLICY}" \
    ttl=1h

Step 2. option a) install the extension for lambda functions packaged in zip archives

If you deploy your Lambda function as a zip file, you can add the extension to your Lambda layers using the console or cli:

arn:aws:lambda:<your-region>:634166935893:layer:vault-lambda-extension:11

Step 2. option b) install the extension for lambda functions packaged in container images

Alternatively, if you deploy your Lambda function as a container image, simply place the built binary in the /opt/extensions directory of your image.

Fetch the binary from releases.hashicorp.com. The following command requires cURL.

$ curl --silent https://releases.hashicorp.com/vault-lambda-extension/0.5.0/vault-lambda-extension_0.5.0_linux_amd64.zip \
  --output vault-lambda-extension.zip

Unzip the downloaded binary.

$ unzip vault-lambda-extension.zip

Optionally, you can verify the integrity of the downloaded zip using the release archive checksum verification instructions here.

Or to build the binary from source. This requires Golang installed. Run from the root of this repository.

$ GOOS=linux GOARCH=amd64 go build -o vault-lambda-extension main.go

Step 3. configure vault-lambda-extension

Configure the extension using Lambda environment variables:

Set the Vault API address.

$ VAULT_ADDR=http://vault.example.com:8200

Set the AWS IAM auth mount point (i.e. the path segment after auth/ from above).

$ VAULT_AUTH_PROVIDER=aws

Set the Vault role to authenticate as. Must be configured for the ARN of your Lambda's role.

$ VAULT_AUTH_ROLE=vault-lambda-role

The path to a secret in Vault. Can be static or dynamic. Unless VAULT_SECRET_FILE is specified, JSON response will be written to /tmp/vault/secret.json.

$ VAULT_SECRET_PATH=secret/lambda-app/token

If everything is correctly set up, your Lambda function can then read secret material from /tmp/vault/secret.json. The exact contents of the JSON object will depend on the secret read, but its schema is the Secret struct from the Vault API module.

Alternatively, you can send normal Vault API requests over HTTP to the local proxy at http://127.0.0.1:8200, and the extension will add authentication before forwarding the request. Vault responses will be returned unmodified. Although local communication is over plain HTTP, the proxy server will use TLS to communicate with Vault if configured to do so as detailed below.

Configuration

The extension is configured via Lambda environment variables. Most of the Vault CLI client's environment variables are available, as well as some additional variables to configure auth, which secret(s) to read and where to write secrets.

The remaining environment variables are not required, and function exactly as described in the Vault Commands (CLI) documentation. However, note that VAULT_CLIENT_TIMEOUT cannot extend the timeout beyond the 10s initialization timeout imposed by the Extensions API when writing files to disk.

AWS STS client configuration

In addition to Vault configuration, you can configure certain aspects of the STS client the extension uses through the usual AWS environment variables. For example, if your Vault instance's IAM auth is configured to use regional STS endpoints:

$ vault write auth/aws/config/client \
     sts_endpoint="https://sts.eu-west-1.amazonaws.com" \
     sts_region="eu-west-1"

Then you may need to configure the extension's STS client to also use the regional STS endpoint by setting AWS_STS_REGIONAL_ENDPOINTS=regional, because both the AWS Golang SDK and Vault IAM auth method default to using the global endpoint in many regions. See documentation on sts_regional_endpoints for more information.

Caching

Caching can be configured for the extension's local proxy server so that it does not forward every HTTP request to Vault. The main consideration behind caching design is to make caching an explicit opt-in at the request level, so that it is only enabled for scenarios where caching makes sense without negative impact in others. To turn on caching, set the environment variable VAULT_DEFAULT_CACHE_TTL to a valid value that is parsable as a time.Duration in Go, for example, "15m", "1h", "2m3s" or "1h2m3s", depending on application needs. An invalid or negative value will be treated the same as a missing value, in which case, caching will not be set up and enabled.

Then requests with HTTP method of "GET", and the HTTP header X-Vault-Cache-Control: cache will be returned directly from the cache if there's a cache hit. On a cache miss the request will be forwarded to Vault and the response returned and cached. If the header is set to X-Vault-Cache-Control: recache, the cache lookup will be skipped, and the request will be forwarded to Vault and the response returned and cached. Currently, the cache key is a hash of the request URL path, headers, body, and token.

Caching may also be enabled for all requests by setting the environment variable VAULT_DEFAULT_CACHE_ENABLED to true. Then all requests will be fetched and/or cached as though the header X-Vault-Cache-Control: cache was present. Setting the header to nocache on a request will opt-out of caching entirely in this configuration. Setting the header to recache will skip the cache lookup and return and cache the response from Vault as described previously.

Limitations

Secrets written to disk or returned from the proxy server will not be automatically refreshed when they expire. This is particularly important if you configure the extension to write secrets to disk, because the extension will only write to disk once per execution environment, rather than once per function invocation. If you use provisioned concurrency or if your Lambda is invoked often enough that execution contexts live beyond the lifetime of the secret, then secrets on disk are likely to become invalid.

In line with Lambda best practices, we recommend avoiding writing secrets to disk where possible, and exclusively consuming secrets via the proxy server. However, the proxy server will still not perform any additional processing with returned secrets such as automatic lease renewal. The proxy server's own Vault auth token is the only thing that gets automatically refreshed. It will synchronously refresh its own token before proxying requests if the token is expired (including a grace window), and it will attempt to renew its token if the token is nearly expired but renewable. The proxy will also immediately refresh its token if the incoming request header X-Vault-Token-Options: revoke is present.

Performance impact

AWS Lambda pricing is based on number of invocations, time of execution and memory used. The following table details some approximate performance related statistics to help assess the cost impact of this extension. Note that AWS Lambda allocates CPU power in proportion to memory so results will vary widely. These benchmarks were run with the minimum 128MB of memory allocated so aim to give an approximate baseline.

Uploading to your own AWS account and region

If you would like to upload the extension as a Lambda layer in your own AWS account and region, you can do the following:

$ curl --silent https://releases.hashicorp.com/vault-lambda-extension/0.5.0/vault-lambda-extension_0.5.0_linux_amd64.zip \
  --output vault-lambda-extension.zip

Set your target AWS region.

$ export REGION="YOUR REGION HERE"

Upload the extension as a Lambda layer.

$ aws lambda publish-layer-version \
     --layer-name vault-lambda-extension \
     --zip-file  "fileb://vault-lambda-extension.zip" \
     --region "${REGION}"

Tutorial

For step-by-step instructions, refer to the Vault AWS Lambda Extension tutorial for details on how to create an AWS Lambda function and use the Vault Lambda Extension to authenticate with Vault.