AWS secrets engine

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are time-based and are automatically revoked when the Vault lease expires.

Vault supports three different types of credentials to retrieve from AWS:

1. iam_user: Vault will create an IAM user for each lease, attach the managed and inline IAM policies as specified in the role to the user, and if a permissions boundary is specified on the role, the permissions boundary will also be attached. Vault will then generate an access key and secret key for the IAM user and return them to the caller. IAM users have no session tokens and so no session token will be returned. Vault will delete the IAM user upon reaching the TTL expiration.
2. assumed_role: Vault will call sts:AssumeRole and return the access key, secret key, and session token to the caller.
3. federation_token: Vault will call sts:GetFederationToken passing in the supplied AWS policy document and return the access key, secret key, and session token to the caller.
4. session_token: Vault will call sts:GetSessionToken and return the access key, secret key, and session token to the caller.

Static roles

The AWS secrets engine supports the concept of "static roles", which are a 1-to-1 mapping of Vault Roles to IAM users. The current password for the user is stored and automatically rotated by Vault on a configurable period of time. This is in contrast to dynamic secrets, where a unique username and password pair are generated with each credential request. When credentials are requested for the Role, Vault returns the current Access Key ID and Secret Access Key for the configured user, allowing anyone with the proper Vault policies to have access to the IAM credentials.

Please see the API documentation for details on this feature.

Setup

Most secrets engines must be configured in advance before they can perform their functions. These steps are usually completed by an operator or configuration management tool.

1. Enable the AWS secrets engine:

   $ vault secrets enable aws
   Success! Enabled the aws secrets engine at: aws/
   

   By default, the secrets engine will mount at the name of the engine. To enable the secrets engine at a different path, use the -path argument.

2. Configure the credentials that Vault uses to communicate with AWS to generate the IAM credentials:

   $ vault write aws/config/root \
       access_key=AKIAJWVN5Z4FOFT7NLNA \
       secret_key=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 \
       region=us-east-1
   

   Internally, Vault will connect to AWS using these credentials. As such, these credentials must be a superset of any policies which might be granted on IAM credentials. Since Vault uses the official AWS SDK, it will use the specified credentials. You can also specify the credentials via the standard AWS environment credentials, shared file credentials, or IAM role/ECS task credentials. (Note that you can't authorize vault with IAM role credentials if you plan on using STS Federation Tokens, since the temporary security credentials associated with the role are not authorized to use GetFederationToken.)

   In some cases, you cannot set sensitive IAM security credentials in your Vault configuration. For example, your organization may require that all security credentials are short-lived or explicitly tied to a machine identity.

   To provide IAM security credentials to Vault, we recommend using Vault plugin workload identity federation (WIF).

   Notice: Even though the path above is aws/config/root, do not use your AWS root account credentials. Instead, generate a dedicated user or role.

3. Alternatively, configure the audience claim value and the role ARN to assume for plugin workload identity federation:

   $ vault write aws/config/root \
       identity_token_audience="<TOKEN AUDIENCE>" \
       role_arn="<AWS ROLE ARN>"
   

   Vault's identity token provider will internally sign the plugin identity token JWT. Given a trust relationship is configured between Vault and AWS via Web Identity Federation, the secrets engine can exchange this identity token to obtain ephemeral STS credentials.

   Notice: For this trust relationship to be established, AWS must have an an IAM OIDC identity provider configured with information about the fully qualified and network-reachable Issuer URL for Vault's plugin identity token provider. This is to ensure that AWS can fetch the JWKS public keys and verify the plugin identity token signature. To configure Vault's Issuer, please refer to the Identity Tokens documentation

4. Configure a Vault role that maps to a set of permissions in AWS as well as an AWS credential type. When users generate credentials, they are generated against this role. An example:

   $ vault write aws/roles/my-role \
       credential_type=iam_user \
       policy_document=-<<EOF
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "ec2:*",
         "Resource": "*"
       }
     ]
   }
   EOF
   

   This creates a role named "my-role". When users generate credentials against this role, Vault will create an IAM user and attach the specified policy document to the IAM user. Vault will then create an access key and secret key for the IAM user and return these credentials. You supply a user inline policy and/or provide references to an existing AWS policy's full ARN and/or a list of IAM groups:

   $ vault write aws/roles/my-other-role \
       policy_arns=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess,arn:aws:iam::aws:policy/IAMReadOnlyAccess \
       iam_groups=group1,group2 \
       credential_type=iam_user \
       policy_document=-<<EOF
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "ec2:*",
         "Resource": "*"
       }
     ]
   }
   EOF
   

   For more information on IAM policies, please see the AWS IAM policy documentation.

Usage

After the secrets engine is configured and a user/machine has a Vault token with the proper permission, it can generate credentials.

1. Generate a new credential by reading from the /creds endpoint with the name of the role:

   $ vault read aws/creds/my-role
   Key                Value
   ---                -----
   lease_id           aws/creds/my-role/f3e92392-7d9c-09c8-c921-575d62fe80d8
   lease_duration     768h
   lease_renewable    true
   access_key         AKIAIOWQXTLW36DV7IEA
   secret_key         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
   session_token     <nil>
   

   Each invocation of the command will generate a new credential.

   Unfortunately, IAM credentials are eventually consistent with respect to other Amazon services. If you are planning on using these credential in a pipeline, you may need to add a delay of 5-10 seconds (or more) after fetching credentials before they can be used successfully.

   If you want to be able to use credentials without the wait, consider using the STS method of fetching keys. IAM credentials supported by an STS token are available for use as soon as they are generated.

2. Rotate the credentials that Vault uses to communicate with AWS:

   $ vault write -f aws/config/rotate-root
   Key           Value
   ---           -----
   access_key    AKIA3ALIVABCDG5XC8H4
   

   Note

   Calls from Vault to AWS may fail immediately after calling aws/config/rotate-root until AWS becomes consistent again. Refer to the AWS secrets engine API reference for additional information on rotating IAM credentials.

IAM permissions policy for Vault

The aws/config/root credentials need permission to manage dynamic IAM users. Here is an example AWS IAM policy that grants the most commonly required permissions Vault needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:AttachUserPolicy",
        "iam:CreateAccessKey",
        "iam:CreateUser",
        "iam:DeleteAccessKey",
        "iam:DeleteUser",
        "iam:DeleteUserPolicy",
        "iam:DetachUserPolicy",
        "iam:GetUser",
        "iam:ListAccessKeys",
        "iam:ListAttachedUserPolicies",
        "iam:ListGroupsForUser",
        "iam:ListUserPolicies",
        "iam:PutUserPolicy",
        "iam:AddUserToGroup",
        "iam:RemoveUserFromGroup",
        "iam:TagUser"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"]
    }
  ]
}

Vault also supports AWS Permissions Boundaries when creating IAM users. If you wish to enforce that Vault always attaches a permissions boundary to an IAM user, you can use a policy like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateAccessKey",
        "iam:DeleteAccessKey",
        "iam:DeleteUser",
        "iam:ListAccessKeys",
        "iam:ListAttachedUserPolicies",
        "iam:ListGroupsForUser",
        "iam:ListUserPolicies",
        "iam:AddUserToGroup",
        "iam:RemoveUserFromGroup"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:AttachUserPolicy",
        "iam:CreateUser",
        "iam:DeleteUserPolicy",
        "iam:DetachUserPolicy",
        "iam:PutUserPolicy"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"],
      "Condition": {
        "StringEquals": {
          "iam:PermissionsBoundary": [
            "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:policy/PolicyName"
          ]
        }
      }
    }
  ]
}

where the "iam:PermissionsBoundary" condition contains the list of permissions boundary policies that you wish to ensure that Vault uses. This policy will ensure that Vault uses one of the permissions boundaries specified (not all of them).

Plugin Workload Identity Federation (WIF)

The AWS secrets engine supports the Plugin WIF workflow, and has a source of identity called a plugin identity token. The plugin identity token is a JWT that is internally signed by Vault's plugin identity token issuer.

If there is a trust relationship configured between Vault and AWS through Web Identity Federation, the secrets engine can exchange its identity token for short-lived STS credentials needed to perform its actions.

Exchanging identity tokens for STS credentials lets the AWS secrets engine operate without configuring explicit access to sensitive IAM security credentials.

To configure the secrets engine to use plugin WIF:

1. Ensure that Vault openid-configuration and public JWKS APIs are network-reachable by AWS. We recommend using an API proxy or gateway if you need to limit Vault API exposure.

2. Create an IAM OIDC identity provider in AWS.

   1. The provider URL must point at your Vault plugin identity token issuer with the /.well-known/openid-configuration suffix removed. For example: https://host:port/v1/identity/oidc/plugins.
   2. The audience should uniquely identify the recipient of the plugin identity token. In AWS, the recipient is the identity provider. We recommend using the host:port/v1/identity/oidc/plugins portion of the provider URL as your recipient since it will be unique for each configured identity provider.

3. Create a web identity role in AWS with the same audience used for your IAM OIDC identity provider.

4. Configure the AWS secrets engine with the IAM OIDC audience value and web identity role ARN.

$ vault write aws/config/root \
    identity_token_audience="vault.example/v1/identity/oidc/plugins" \
    role_arn="arn:aws:iam::123456789123:role/example-web-identity-role"

Your secrets engine can now use plugin WIF for its configuration credentials. By default, WIF credentials have a time-to-live of 1 hour and automatically refresh when they expire.

Please see the API documentation for more details on the fields associated with plugin WIF.

STS credentials

The above demonstrated usage with iam_user credential types. As mentioned, Vault also supports assumed_role, federation_token, and session_token credential types.

STS federation tokens

An STS federation token inherits a set of permissions that are the combination (intersection) of four sets of permissions:

1. The permissions granted to the aws/config/root credentials
2. The user inline policy configured in the Vault role
3. The managed policy ARNs configured in the Vault role
4. An implicit deny policy on IAM or STS operations.

Roles with a credential_type of federation_token can specify one or more of the policy_document, policy_arns, and iam_groups parameters in the Vault role.

The aws/config/root credentials require IAM permissions for sts:GetFederationToken and the permissions to delegate to the STS federation token. For example, this policy on the aws/config/root credentials would allow creation of an STS federated token with delegated ec2:* permissions (or any subset of ec2:* permissions):

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "ec2:*",
      "sts:GetFederationToken"
    ],
    "Resource": "*"
  }
}

An ec2_admin role would then assign an inline policy with the same ec2:* permissions.

$ vault write aws/roles/ec2_admin \
    credential_type=federation_token \
    policy_document=@policy.json

The policy.json file would contain an inline policy with similar permissions, less the sts:GetFederationToken permission. (We could grant sts:GetFederationToken permissions, but STS attaches attach an implicit deny that overrides the allow.)

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "ec2:*",
    "Resource": "*"
  }
}

To generate a new set of STS federation token credentials, we simply write to the role using the aws/sts endpoint:

$ vault write aws/sts/ec2_admin ttl=60m
Key             Value
lease_id        aws/sts/ec2_admin/31d771a6-fb39-f46b-fdc5-945109106422
lease_duration  60m0s
lease_renewable false
access_key      ASIAJYYYY2AA5K4WIXXX
secret_key      HSs0DYYYYYY9W81DXtI0K7X84H+OVZXK5BXXXX
session_token   AQoDYXdzEEwasAKwQyZUtZaCjVNDiXXXXXXXXgUgBBVUUbSyujLjsw6jYzboOQ89vUVIehUw/9MreAifXFmfdbjTr3g6zc0me9M+dB95DyhetFItX5QThw0lEsVQWSiIeIotGmg7mjT1//e7CJc4LpxbW707loFX1TYD1ilNnblEsIBKGlRNXZ+QJdguY4VkzXxv2urxIH0Sl14xtqsRPboV7eYruSEZlAuP3FLmqFbmA0AFPCT37cLf/vUHinSbvw49C4c9WQLH7CeFPhDub7/rub/QU/lCjjJ43IqIRo9jYgcEvvdRkQSt70zO8moGCc7pFvmL7XGhISegQpEzudErTE/PdhjlGpAKGR3d5qKrHpPYK/k480wk1Ai/t1dTa/8/3jUYTUeIkaJpNBnupQt7qoaXXXXXXXXXX

STS Session Tokens

The session_token credential type is used to generate short-lived credentials under the root config. To create these with Vault and AWS, you must configure Vault to use IAM user credentials. AWS does not allow temporary credentials, like those from an IAM instance profile, to be used when generating session tokens.

$ vault write aws/roles/temp_user \
    credential_type=session_token

To generate a new set of STS federation token credentials, write to the temp_user role using the aws/creds endpoint:

$ vault read aws/sts/temp_user ttl=60m
Key             Value
lease_id        aws/creds/temp_user/w4eKbMaJOi1xLqG3MWk7y8n6
lease_duration  60m0s
lease_renewable false
access_key      ASIAJYYYY2AA5K4WIXXX
secret_key      HSs0DYYYYYY9W81DXtI0K7X84H+OVZXK5BXXXX
session_token   AQoDYXdzEEwasAKwQyZUtZaCjVNDiXXXXXXXXgUgBBVUUbSyujLjsw6jYzboOQ89vUVIehUw/9MreAifXFmfdbjTr3g6zc0me9M+dB95DyhetFItX5QThw0lEsVQWSiIeIotGmg7mjT1//e7CJc4LpxbW707loFX1TYD1ilNnblEsIBKGlRNXZ+QJdguY4VkzXxv2urxIH0Sl14xtqsRPboV7eYruSEZlAuP3FLmqFbmA0AFPCT37cLf/vUHinSbvw49C4c9WQLH7CeFPhDub7/rub/QU/lCjjJ43IqIRo9jYgcEvvdRkQSt70zO8moGCc7pFvmL7XGhISegQpEzudErTE/PdhjlGpAKGR3d5qKrHpPYK/k480wk1Ai/t1dTa/8/3jUYTUeIkaJpNBnupQt7qoaXXXXXXXXXX

Session tokens may also require an MFA-based TOTP to be provided if the IAM user is configured to require it. If so, the Vault role requires the MFA device serial number to be set, and the TOTP may be provided when reading credentials from the Vault role.

$ vault write aws/roles/mfa_user \
    credential_type=session_token \
    mfa_serial_number="arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:mfa/device-name"

$ vault read aws/creds/mfa_user mfa_code=123456

STS AssumeRole

The assumed_role credential type is typically used for cross-account authentication or single sign-on (SSO) scenarios. In order to use an assumed_role credential type, you must configure outside of Vault:

1. An IAM role
2. IAM inline policies and/or managed policies attached to the IAM role
3. IAM trust policy attached to the IAM role to grant privileges for Vault to assume the role

assumed_role credentials offer a few benefits over federation_token:

1. Assumed roles can invoke IAM and STS operations, if granted by the role's IAM policies.
2. Assumed roles support cross-account authentication
3. Temporary credentials (such as those granted by running Vault on an EC2 instance in an IAM instance profile) can retrieve assumed_role credentials (but cannot retrieve federation_token credentials).

The aws/config/root credentials must be allowed sts:AssumeRole through one of two methods:

1. The credentials have an IAM policy attached to them against the target role:

   {
     "Version": "2012-10-17",
     "Statement": {
       "Effect": "Allow",
       "Action": "sts:AssumeRole",
       "Resource": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:role/RoleNameToAssume"
     }
   }
   

2. A trust policy is attached to the target IAM role for the principal:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/VAULT-AWS-ROOT-CONFIG-USER-NAME"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

When specifying a Vault role with a credential_type of assumed_role, you can specify more than one IAM role ARN. If you do so, Vault clients can select which role ARN they would like to assume when retrieving credentials from that role.

Further, you can specify both a policy_document and policy_arns parameters; if specified, each acts as a filter on the IAM permissions granted to the assumed role. If iam_groups is specified, the inline and attached policies for each IAM group will be added to the policy_document and policy_arns parameters, respectively, when calling sts:AssumeRole. For an action to be allowed, it must be permitted by both the IAM policy on the AWS role that is assumed, the policy_document specified on the Vault role (if specified), and the managed policies specified by the policy_arns parameter. (The policy_document parameter is passed in as the Policy parameter to the sts:AssumeRole API call, while the policy_arns parameter is passed in as the PolicyArns parameter to the same call.)

Note: When multiple role_arns are specified, clients requesting credentials can specify any of the role ARNs that are defined on the Vault role in order to retrieve credentials. However, when policy_document, policy_arns, or iam_groups are specified, that will apply to ALL role credentials retrieved from AWS.

Let's create a "deploy" policy using the arn of our role to assume:

$ vault write aws/roles/deploy \
    role_arns=arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:role/RoleNameToAssume \
    credential_type=assumed_role

To generate a new set of STS assumed role credentials, we again write to the role using the aws/sts endpoint:

$ vault write aws/sts/deploy ttl=60m
Key             Value
lease_id        aws/sts/deploy/31d771a6-fb39-f46b-fdc5-945109106422
lease_duration  60m0s
lease_renewable false
access_key      ASIAJYYYY2AA5K4WIXXX
secret_key      HSs0DYYYYYY9W81DXtI0K7X84H+OVZXK5BXXXX
session_token   AQoDYXdzEEwasAKwQyZUtZaCjVNDiXXXXXXXXgUgBBVUUbSyujLjsw6jYzboOQ89vUVIehUw/9MreAifXFmfdbjTr3g6zc0me9M+dB95DyhetFItX5QThw0lEsVQWSiIeIotGmg7mjT1//e7CJc4LpxbW707loFX1TYD1ilNnblEsIBKGlRNXZ+QJdguY4VkzXxv2urxIH0Sl14xtqsRPboV7eYruSEZlAuP3FLmqFbmA0AFPCT37cLf/vUHinSbvw49C4c9WQLH7CeFPhDub7/rub/QU/lCjjJ43IqIRo9jYgcEvvdRkQSt70zO8moGCc7pFvmL7XGhISegQpEzudErTE/PdhjlGpAKGR3d5qKrHpPYK/k480wk1Ai/t1dTa/8/3jUYTUeIkaJpNBnupQt7qoaXXXXXXXXXX

Troubleshooting

Dynamic IAM user errors

If you get an error message similar to either of the following, the root credentials that you wrote to aws/config/root have insufficient privilege:

$ vault read aws/creds/deploy
* Error creating IAM user: User: arn:aws:iam::000000000000:user/hashicorp is not authorized to perform: iam:CreateUser on resource: arn:aws:iam::000000000000:user/vault-root-1432735386-4059

$ vault revoke aws/creds/deploy/774cfb27-c22d-6e78-0077-254879d1af3c
Revoke error: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/revoke/aws/creds/deploy/774cfb27-c22d-6e78-0077-254879d1af3c
Code: 400. Errors:

* invalid request

If you get stuck at any time, simply run vault path-help aws or with a subpath for interactive help output.

STS federated token errors

Vault generates STS tokens using the IAM credentials passed to aws/config.

Those credentials must have two properties:

• They must have permissions to call sts:GetFederationToken.
• The capabilities of those credentials have to be at least as permissive as those requested by policies attached to the STS creds.

If either of those conditions are not met, a "403 not-authorized" error will be returned.

See http://docs.aws.amazon.com/STS/latest/APIReference/API_GetFederationToken.html for more details.

Vault 0.5.1 or later is recommended when using STS tokens to avoid validation errors for exceeding the AWS limit of 32 characters on STS token names.

AWS instance metadata timeouts

Anytime Vault uses the instance metadata service on an EC2 instance, such as for getting credentials from the instance profile, there may be a delay with the introduction of v2 of the instance metadata service (IMDSv2). The AWS SDK used by Vault first attempts to connect to IMDSv2, and if that times out, it falls back to v1. In Vault 1.4, this timeout can take up to 2 minutes. In Vault 1.5.5 and later, it can take up to 2 seconds with this fix: #10133.

The timeout occurs in situations where there is a proxy between Vault and IMDSv2, and the instance hop limit is set to less than the number of "hops" between Vault and IMDSv2. For example, if Vault is running in docker on an EC2 instance with the instance hop limit set to 1, the AWS SDK client will attempt to connect to IMDSv2, timeout, and fall back to IMDSv1 because of the extra network hop between docker and IMDS.

To avoid the timeout behavior, the hop limit may be adjusted on the underlying EC2 instances. With the docker example, setting the hop limit to 2 will allow the AWS SDK in Vault to connect to IMDSv2 without delay.

API

The AWS secrets engine has a full HTTP API. Please see the AWS secrets engine API for more details.