Administer Consul access control tokens with Vault

HashiCorp's Vault has a secrets engine for generating short-lived Consul tokens, the Consul Secrets Engine. This engine generates Consul Access Control (ACL) tokens dynamically based on Consul ACL policies.

In this tutorial, you will configure the Consul secrets engine in Vault to generate Consul tokens.

The integration with Vault's Consul secrets engine has the following workflow,

1. create a Consul ACL policy to define tokens' privileges
2. create a Vault role to map the policy
3. create a token with Vault
4. verify that the token got synced with Consul

Before using this workflow, you must bootstrap the Consul ACL system and configure the Vault's Consul secrets engine, the tutorial provides you with steps for both requirements.

Prerequisites

• Consul: to complete this tutorial, you can use a local Consul dev agent or an existing Consul deployment.
• Vault: this tutorial assumes you already have a running Vault cluster in your network. You can use a local Vault dev server or an existing Vault deployment.

The diagram below shows the minimal architecture needed to demonstrate the functionality.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Configure Consul ACL

As a prerequisite for the integration you must bootstrap the Consul ACL system in your datacenter.

ACL configuration file

To be able to configure Consul tokens and policies, you will need to enable ACLs in your Consul datacenter using a configuration similar to the following:

# ACL configuration
acl = {
    enabled = true
    default_policy = "deny"
    enable_token_persistence = true
}

Verify that the Consul server started correctly by checking the logs. You should get a log message like the following when ACLs are correctly enabled:

agent.server: initializing acls

Bootstrap the ACL system

First, configure your environment to be able to interact with the Consul agent.

$ export CONSUL_HTTP_ADDR=localhost:8500

Then, bootstrap the ACL system to start using tokens.

$ consul acl bootstrap | tee consul.bootstrap

AccessorID:       e57b446b-2da0-bce2-f01c-6c0134d8e19b
SecretID:         bba19e7c-9f47-2b08-f0ea-e1bca43ba9c5
Description:      Bootstrap Token (Global Management)
Local:            false
Create Time:      2020-02-20 17:01:13.105174927 +0000 UTC
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Finally, set the token for the command using the CONSUL_HTTP_TOKEN environment variable.

$ export CONSUL_HTTP_TOKEN=$(cat consul.bootstrap | grep SecretID | awk '{print $2}')

You can now use the bootstrap token to create other ACL policies and tokens for the rest of your datacenter. You will also use the same token to configure the Vault secrets engine.

Configure the Consul secrets engine in Vault

In order to communicate with the Vault server, you will need to set the address and token.

$ export VAULT_ADDR='http://127.0.0.1:8200'

$ export VAULT_TOKEN="root"

Enable secrets engine

The Consul secrets engine must be enabled before it can perform its functions.

$ vault secrets enable consul

Success! Enabled the consul secrets engine at: consul/

Configure secrets engine access

Once you have the Consul secrets engine enabled, configure access with Consul's address and management token.

$ vault write consul/config/access address=${CONSUL_HTTP_ADDR} token=${CONSUL_HTTP_TOKEN}

Success! Data written to: consul/config/access

Create a Consul server policy

Once the ACL system is initialized, you should still observe warnings in your Consul logs similar to the following example.

 [WARN]  agent: Node info update blocked by ACLs: node=c19d6af9-760b-a3cc-bbdd-b4f7209a79de accessorID=00000000-0000-0000-0000-000000000002
 [WARN]  agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002

These warnings indicate that:

• the agent is trying to update its info in your Consul datacenter, but has been denied by the ACL system because it does not have the correct permissions.
• the agent does not have an assigned token to perform the request (because 00000000-0000-0000-0000-000000000002 is the anonymous policy used when no token is presented)

The first step towards a more fine grained ACL approach is to create individual tokens for the server agents so that they can interact properly with the rest of the Consul datacenter without being assigned an administrative token.

For this tutorial, you will create a policy that permits server agents to register themselves. The servers will follow the naming pattern server-<count_or_id>, and have the necessary permissions to locate other agents and discover services.

This policy will be used by Vault when creating tokens for Consul.

Create a file named server_policy.hclwith the following content.

node_prefix "server-" {
  policy = "write"
}
node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}

Create the policy with the consul acl command.

$ consul acl policy create \
  -name consul-servers \
  -rules @server_policy.hcl

Once the policy is created you need to associate it to a token in order to use it.

Create a Vault server role

In the previous step, you created a policy named consul-servers to define the server agents' permissions.

You are going to use that policy when creating tokens for Consul servers.

If you want to generate numerous tokens with identical policies we recommend you associate each policy with a common role.

Roles allow you to group a set of policies and service identities into a reusable higher-level entity that can be applied to many tokens.

For this tutorial, you are going to configure a role that maps a name in Vault to a set of Consul ACL policies. If users generate credentials associated with this role, they will be associated with the policies of that role.

$ vault write consul/roles/consul-server-role policies=consul-servers

Success! Data written to: consul/roles/consul-server-role

Obtain and test a Consul ACL token

Create token from Vault

Create a Consul token using the existing Vault token.

$ vault read consul/creds/consul-server-role | tee consul.server

Key                Value
---                -----
lease_id           consul/creds/consul-server-role/bHDVG24vCO8BJ90ONjxrbKP6
lease_duration     768h
lease_renewable    true
accessor           2b9bb86a-d632-ea28-3890-7d1e0690ff57
local              false
token              5ea5eadc-807d-68c2-e47f-a1ac37e906b7

Verify token was created in Consul

Verify that the token is created correctly in Consul by looking it up by its accessor.

$ export CONSUL_SERVER_ACCESSOR=$(cat consul.server | grep accessor | awk '{print $2}')

$ consul acl token read -id ${CONSUL_SERVER_ACCESSOR}

AccessorID:       2b9bb86a-d632-ea28-3890-7d1e0690ff57
SecretID:         5ea5eadc-807d-68c2-e47f-a1ac37e906b7
Description:      Vault consul-server-role token 1598547759570180867
Local:            false
Create Time:      2020-08-27 17:02:39.572006543 +0000 UTC
Policies:
   6fa2c574-6951-51db-310a-672e328f2aba - consul-servers

Any user or process with access to Vault can now create short lived Consul tokens in order to carry out operations, thus centralizing the management of Consul tokens.

Understand the differences in token representations

The way Vault and Consul refer to tokens in the command output is slightly different. The following table expresses the relationship between the two outputs.

Using the Consul secrets engine with Vault ensures that these values are kept consistent when the tokens are replicated to Consul.

Set Consul server agent token

For this tutorial you saved the token inside the consul.server file.

You can retrieve it and store it in an environment variable.

$ export CONSUL_SERVER_TOKEN=$(cat consul.server | grep token | awk '{print $2}')

Finally, register the token with the server as its agent token.

$ consul acl set-agent-token agent $(cat consul.server | grep token | awk '{print $2}')

ACL token "agent" set successfully

Once the token is applied, check the Consul logs once more and verify that the warning lines are not being logged anymore.

...
[INFO]  agent: Updated agent's ACL token: token=agent
[INFO]  agent: Synced node info
...

Token lease duration and rotation

Using Vault's secret engine you can uniform the process of token creation using Vault but you will still need to take care of token rotation inside your Consul datacenter.

The tokens created using the integration with Vault's Consul secrets engine are created with a Time To Live (TTL) of 30 days.

You can verify the lease duration and expiration time on Vault UI:

At this moment this value is not configurable, the recommended approach for operators is to rotate the tokens by creating a new token using:

$ vault read consul/creds/consul-server-role

and then, once the token is synchronized with Consul apply the token to the agents using the consul acl command as showed earlier in this tutorial.

Rotating tokens is a core activity for keeping your Consul datacenter secured against secrets leaking. You should consider defining an internal policy that regulates the rotation interval and make sure is respected.

Next steps

In this tutorial, you configured Vault's Consul secrets engine so that you could generate Consul tokens using Vault.

To continue securing your datacenter with Vault, you can follow Generate mTLS Certificates for Consul using Vault tutorial.

To learn how to configure gossip encryption for Consul using Vault for secrets storage and management, refer to the Secure Consul gossip communication with Vault tutorial.