HashiConf 2024 Now streaming live from Boston! Attend for free
Consul
Archive

Automate Consul agent security with auto config

  • 18min
  • |
  • Vault
  • Consul

Consul auto_config is a highly scalable method used to distribute secure properties such as Access Control List (ACL) tokens, TLS certificates, gossip encryption keys, and other configuration settings to all Consul agents in a datacenter. This can provide a great boost in usability for securing agents against eavesdropping, tampering, and spoofing without having to manually distribute or rotate secrets across the datacenter. Additionally, auto_config reduces the technical overhead associated with securing Consul agents.

In this tutorial, you will:

  • Create a secure local Consul datacenter using Docker Compose
  • Use the created environment to inspect the server and client auto_config settings
  • Generate a JSON web token (JWT) with either Vault or secint
  • Securely join a client to the Consul datacenter with the auto_config method

JSON web tokens (JWTs) are an open, industry standard method for representing claims securely between two parties. With the auto_config method, Consul clients use JWTs to securely retrieve gossip encryption keys, TLS certificates, and/or other security setting changes from Consul servers. Since tokens are credentials, great care must be taken to prevent security issues. In general, you should not keep tokens longer than required. To learn more about JSON web tokens, check the JWT documentation page.

While this tutorial uses elements that are not suitable for production environments including example certificates, example gossip encryption keys, and global ACL token usage, it will teach you the core concepts for deploying and interacting with a secure Consul datacenter using auto_config. Refer to the Consul Reference Architecture for Consul best practices and the Docker Documentation for Docker best practices.

Prerequisites

This tutorial builds on the concepts learned in the Consul getting started collection and the Deploy a secure local datacenter with Docker Compose tutorial.

Docker

You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker on your specific operating system here. Docker Engine 20.10.7 was used in this tutorial.

Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the Docker Compose install guide for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.

Choose your learning path

You can utilize the auto_config workflow with HashiCorp Vault or a third-party production-grade secrets management platform. In this tutorial we will demonstrate the Vault workflow, and a generic workflow with a proof-of-concept tool, secint.

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp/learn-consul-docker.git

Change into the directory with the newly cloned repository. This directory contains the complete configuration files.

$ cd learn-consul-docker/datacenter-deploy-auto-config/vault

Check out the v0.3 tag of the repository.

$ git checkout v0.3

Inspect configuration files

You will use the following resources to deploy five containers; three Consul servers, a Vault server, and a single Consul client.

Docker Compose YAML

The following Docker Compose configuration file instructs Docker to create one Vault server and four Consul containers using the respective configuration files. These files will configure various container settings and bootstrap the Consul datacenter with three secure Consul servers. Three servers in a datacenter is the recommended minimum for achieving a balance between availability and performance. These servers together run the Raft-driven consistent state store for updating catalog, session, prepared query, ACLs, and KV state.

Inside your working directory, inspect the file named docker-compose.yml.

datacenter-deploy-auto-config/vault/docker-compose.yml

version: '3.7'

services:
  
  consul-server1:
    image: hashicorp/consul:1.11.2
    container_name: consul-server1
    hostname: consul-server1
    depends_on: 
      - vault-server
    restart: always
    volumes:
     - ./consul/server1.json:/consul/config/server1.json
     - ./certs/:/consul/config/certs/:ro
    networks:
      - hashicorp
    ports:
      - "8500:8500"
      - "8600:8600/tcp"
      - "8600:8600/udp"
    command: "agent -bootstrap-expect=3"

  consul-server2:
    image: hashicorp/consul:1.11.2
    container_name: consul-server2
    hostname: consul-server2
    depends_on: 
      - vault-server
    restart: always
    volumes:
     - ./consul/server2.json:/consul/config/server2.json
     - ./certs/:/consul/config/certs/:ro
    networks:
      - hashicorp
    command: "agent -bootstrap-expect=3"

  consul-server3:
    image: hashicorp/consul:1.11.2
    container_name: consul-server3
    hostname: consul-server3
    depends_on: 
      - vault-server
    restart: always
    volumes:
     - ./consul/server3.json:/consul/config/server3.json
     - ./certs/:/consul/config/certs/:ro
    networks:
      - hashicorp
    command: "agent -bootstrap-expect=3"

  consul-client:
    image: hashicorp/consul:1.11.2
    container_name: consul-client
    hostname: consul-client
    restart: always
    volumes:
     - ./consul/client.json:/consul/config/client.json
     - ./certs/:/consul/config/certs/:ro
     - ./tokens/:/consul/config/tokens/
    networks:
      - hashicorp
    command: "agent"

  vault-server:
    image: hashicorp/vault:1.8.1
    container_name: vault-server
    hostname: vault-server
    restart: always
    ports:
      - "8200:8200"
    environment:
      VAULT_ADDR: "http://vault-server:8200"
      VAULT_API_ADDR: "http://vault-server:8200"
      VAULT_DEV_ROOT_TOKEN_ID: "vault-plaintext-root-token" 
      CONSUL_HTTP_ADDR: "consul-server1:8500"
      CONSUL_HTTP_TOKEN: "e95b599e-166e-7d80-08ad-aee76e7ddf19"
    cap_add:
      - IPC_LOCK
    volumes:
     - ./vault/policy.json:/vault/policies/policy.json
    networks:
      - hashicorp

networks:
  hashicorp:
    driver: bridge

For more information on the Consul Docker image, check Consul's Docker Hub page.

Consul server configuration

On a Consul server agent with the authorization subkey set to enabled, the server will begin to process client auto_config requests.

Inside your working directory, inspect the three server#.json files.

datacenter-deploy-auto-config/vault/consul/server1.json

{
    "node_name": "consul-server1",
    "server": true,
    "ui_config": {
        "enabled" : true
    },
    "data_dir": "/consul/data",
    "addresses": {
        "http" : "0.0.0.0"
    },
    "retry_join":[
        "consul-server2",
        "consul-server3"
    ],
    "acl": { 
        "enabled": true,
        "default_policy": "deny",
        "enable_token_persistence": true,
        "tokens": {
            "initial_management": "e95b599e-166e-7d80-08ad-aee76e7ddf19",
            "agent": "e95b599e-166e-7d80-08ad-aee76e7ddf19"
        }
    },
    "connect": { "enabled": true },
    "auto_config": {
        "authorization": {
            "enabled": true,
            "static": {
                "oidc_discovery_url": "http://vault-server:8200/v1/identity/oidc",
                "bound_issuer": "http://vault-server:8200/v1/identity/oidc",
                "bound_audiences": ["consul-cluster-dc1"],
                "claim_mappings": {
                    "/consul/hostname": "node_name"
                },
                "claim_assertions": [
                    "value.node_name == \"${node}\""
                ]
            }
        }
    },
    "encrypt": "aPuGh+5UDskRAbkLaXRzFoSOcSM+5vAK+NEYOWHJH7w=",
    "verify_incoming": true,
    "verify_outgoing": true,
    "verify_server_hostname": true,
    "ca_file": "/consul/config/certs/consul-agent-ca.pem",
    "cert_file": "/consul/config/certs/dc1-server-consul-0.pem",
    "key_file": "/consul/config/certs/dc1-server-consul-0-key.pem"
}

Each Consul server configuration file uses the following auto_config options:

  • authorization - Setting the sub key enabled to true enables the authorization service on this agent. This allows the server agent to process auto_config RPC requests from clients.

  • static - This object contains all static authorizer configuration settings.

    • oidc_discovery_url - The URL used to validate JSON web tokens (JWTs). In this example, Vault's OIDC URL endpoint http://vault-server:8200/v1/identity/oidc is used.
    • bound_issuer - The value for matching the iss value in a JSON web token (JWT). The issuer iss claim in a JWT is meant to refer to the resource that issued the JWT. In this example, the Vault server assigns its OIDC URL endpoint http://vault-server:8200/v1/identity/oidc as the issuer for generated JWTs.
    • bound_audiences - The value for matching the aud field of the JSON web token (JWT). The audience aud claim in a JWT is meant to refer to the authorization servers that should accept the token. In this example, Vault assigns the generated JWT tokens with an aud value of consul-cluster-dc1.
    • claim_assertions - List of assertions about the mapped claims required to authorize the incoming RPC request. In this example, "value.node_name == \"${node}\"" sets the value of the node_name variable to the hostname of the consul client making a request to the auto_config authorization servers.
    • claim_mappings - Mappings of claims (key) that will be copied to a metadata field (value). In this example, "/consul/hostname": "node_name" checks the value of the node_name variable against the metadata value of /consul/hostname.

Check the Consul configuration documentation to learn more.

Consul client configuration

On a Consul client agent with auto_config enabled, the client agent will use the value of a JSON web token (JWT) within intro_token_file to communicate with the configured server_addresses to request secure configuration settings. These settings are then merged into any existing configuration on the client agent.

Inside your working directory, inspect the client.json file.

datacenter-deploy-auto-config/vault/consul/client.json

{
    "node_name": "consul-client",
    "data_dir": "/consul/data",
    "ports": {"https":8501},
    "auto_config":{
        "enabled": true,
        "intro_token_file": "/consul/config/tokens/jwt",
        "server_addresses":[
            "consul-server1",
            "consul-server2",
            "consul-server3"
        ]
    },
    "verify_incoming": false,
    "verify_outgoing": true,
    "verify_server_hostname": true,
    "ca_file": "/consul/config/certs/consul-agent-ca.pem"
}

Each Consul client configuration file uses the following auto_config options:

  • enabled - Setting this key to true enables the auto_config client service on the agent. Enabling this option also turns on Consul service mesh because it is required for auto_config to issue certificates to client agents using the Consul service mesh CA.

  • intro_token_file - This specifies the file that contains the JSON web token (JWT) to use for the initial auto_config RPC to the Consul servers.

  • server_addresses - This specifies the addresses of servers in the local datacenter to use for the initial RPC. These addresses support Cloud Auto-Joining and can optionally include a port to use when making the outbound connection. If not port is provided the server RPC port will be used.

Check the Consul configuration documentation to learn more.

Create the Consul datacenter

From within your working directory, run the following Docker Compose command to create your secure local Consul datacenter.

$ docker-compose up --detach

Creating network "vault_hashicorp" with driver "bridge"
Creating consul-client ... done
Creating vault-server  ... done
Creating consul-server3 ... done
Creating consul-server1 ... done
Creating consul-server2 ... done

Note

Your first run will take the longest as Docker will first pull the respective images from the Docker Hub repository. Additional runs will not require to download the image again and should only take a few seconds to complete.

View the Consul UI

Navigate to http://localhost:8500 on your browser to access the Consul UI. Login with the pre-configured ACL token e95b599e-166e-7d80-08ad-aee76e7ddf19.

Click on the "Nodes" option in the top navigation bar to go to the nodes page. Notice that the three consul-server# nodes are present in the datacenter, however, the consul-client node is not present since it cannot join the secure datacenter.

Image of Nodes page in Consul UI.

Retrieve Consul client logs

Check the container logs for consul-client to see details about why the client could not join the Consul datacenter.

$ docker container logs consul-client

[INFO]  agent.auto_config: retrieving initial agent auto configuration remotely
[ERROR] agent.auto_config: intro_token_file did not contain any token

This type of behavior occurs when a client's JSON web token (JWT) cannot be validated by the authorization server. In this example the token file is empty.

Note

Feel free to explore the jwt.io documentation for more information on the properties of a JSON web token (JWT).

Configure Vault to generate JWTs

Vault's built-in Identity Secrets Engine can be used to generate the JSON Web Tokens (JWTs) required to validate Consul client auto_join requests.

Open an interactive shell to the Vault server container.

$ docker exec -it vault-server /bin/sh

/ #

Login to Vault with the pre-configured Vault's root token, vault-plaintext-root-token.

$ vault login

Token (will be hidden):

Named keys are used by a role to sign JSON web tokens (JWTs). The value for allowed_client_ids here will become the value of aud when a JWT is generated. The audience aud claim in a JWT is meant to refer to the authorization servers that should accept the token, in this case, consul-cluster-dc1.

Create a named key.

$ vault write identity/oidc/key/oidc-key-1 allowed_client_ids="consul-cluster-dc1"

Success! Data written to: identity/oidc/key/oidc-key-1

JSON web tokens (JWTs) are generated against a role and signed against a named key. The template='{"consul": {"hostname": "consul-client" } }' will create additional JWT metadata that will be used by the Consul authorization servers to validate the request.

Create a role.

$ vault write identity/oidc/role/oidc-role-1 ttl=12h key="oidc-key-1" client_id="consul-cluster-dc1" template='{"consul": {"hostname": "consul-client" } }'

Success! Data written to: identity/oidc/role/oidc-role-1

Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. In this example, a policy file will be used that only grants a user permission to read the token at the path identity/oidc/token/oidc-role-1.

datacenter-deploy-auto-config/vault/vault/policies/policy.json

{
    "path": {
      "identity/oidc/token/oidc-role-1": {
        "policy": "read"
      }
    }
}

Create a policy using the included policy.json file.

$ vault policy write oidc-policy /vault/policies/policy.json

Success! Uploaded policy: oidc-policy

Note

Feel free to explore Vault's identity token documentation to learn more about identity token attributes, roles, and keys.

Generate a JWT with Vault

An authorized Vault user/application can request a token that encapsulates identity information for their associated entity. These tokens are signed JSON web tokens (JWTs) following the OIDC ID token structure. In this example, a user entity will be created to manually generate a JWT.

Enable Vault's username/password secrets engine.

$ vault auth enable userpass

Success! Enabled userpass auth method at: userpass/

Create a user example with the oidc-policy attached.

$ vault write auth/userpass/users/example password=password policies=oidc-policy

Success! Data written to: auth/userpass/users/example

Login as the user example with the password password.

$ vault login -method=userpass username=example

Password (will be hidden):

After login you should receive an output similar to the following:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.8mqE8WBErCFUfEnZxyiTwkny
token_accessor         x0FS1wkewLLD5D2B4FUvnBRA
token_duration         768h
token_renewable        true
token_policies         ["default" "oidc-policy"]
identity_policies      []
policies               ["default" "oidc-policy"]
token_meta_username    example

Generate a signed JSON web token (JWT). Note that each time this operation is performed, a new JWT will be generated.

$ vault read identity/oidc/token/oidc-role-1

Key          Value
---          -----
client_id    consul-cluster-dc1
token        eyJhbGciOiJSUzI1NiIsImtpZCI6IjI4YjA2NDlmLTdlNjktMWFhMC03ZmYyLWI4ZDU5NGJhZmE5MCJ9.eyJhdWQiOiJjb25zdWwtY2x1c3Rlci1kYzEiLCJjb25zdWwiOnsiaG9zdG5hbWUiOiJjb25zdWwtY2xpZW50In0sImV4cCI6MTYyOTc5MDYwNSwiaWF0IjoxNjI5NzQ3NDA1LCJpc3MiOiJodHRwOi8vdmF1bHQtc2VydmVyOjgyMDAvdjEvaWRlbnRpdHkvb2lkYyIsIm5hbWVzcGFjZSI6InJvb3QiLCJzdWIiOiI4NWE5ZWMxYi1iMTcyLWU1YWEtZmU3Ni0xMzFkOWFjZmVjZTgifQ.eFDQ_TReNvKeMS4si92oiPOBcRbv0bGuVKq4Qns0ObnNrwFWVvDB9HLkCP7VzRuO9l9a3Jzl-Uk__Y_fF_JgWk7s2iZTg9RbBZD0TYz5-ziHU13wd7Onx9OjXjmw-5ah96dDFh3nqkuXJpV9upmVfXA7Zb5goYyfULa7gWeSNPjyjYNx2oirsFwH_xm9No9lttEA33XOGyAGi9UNBlvKdw6uXJfhnWTG2NxEt9y7JO_wNxjXKOUxhVhbb6ZxRZT_enbib1g_b-BVrNvTqB5UfSKyz6h3musoqDsAcLOEeAkl6dfWv3IezhqY2vNm5mQ3lH83AK6dZdwvVcG0DmdIpQ
ttl          12h

Copy the value of token to your clipboard.

Exit the docker terminal to return to your system shell.

$ exit

Note

Feel free to explore the contents of your JSON web token (JWT) using the decoder at jwt.io.

Configure your client with the JWT

Copy the token value to the /vault/tokens/jwt file in your working directory, then save the changes.

datacenter-deploy-auto-config/vault/tokens/jwt

eyJhbGciOiJSUzI1NiIsImtpZCI6IjI4YjA2NDlmLTdlNjktMWFhMC03ZmYyLWI4ZDU5NGJhZmE5MCJ9.eyJhdWQiOiJjb25zdWwtY2x1c3Rlci1kYzEiLCJjb25zdWwiOnsiaG9zdG5hbWUiOiJjb25zdWwtY2xpZW50In0sImV4cCI6MTYyOTc5MDYwNSwiaWF0IjoxNjI5NzQ3NDA1LCJpc3MiOiJodHRwOi8vdmF1bHQtc2VydmVyOjgyMDAvdjEvaWRlbnRpdHkvb2lkYyIsIm5hbWVzcGFjZSI6InJvb3QiLCJzdWIiOiI4NWE5ZWMxYi1iMTcyLWU1YWEtZmU3Ni0xMzFkOWFjZmVjZTgifQ.eFDQ_TReNvKeMS4si92oiPOBcRbv0bGuVKq4Qns0ObnNrwFWVvDB9HLkCP7VzRuO9l9a3Jzl-Uk__Y_fF_JgWk7s2iZTg9RbBZD0TYz5-ziHU13wd7Onx9OjXjmw-5ah96dDFh3nqkuXJpV9upmVfXA7Zb5goYyfULa7gWeSNPjyjYNx2oirsFwH_xm9No9lttEA33XOGyAGi9UNBlvKdw6uXJfhnWTG2NxEt9y7JO_wNxjXKOUxhVhbb6ZxRZT_enbib1g_b-BVrNvTqB5UfSKyz6h3musoqDsAcLOEeAkl6dfWv3IezhqY2vNm5mQ3lH83AK6dZdwvVcG0DmdIpQ

Delete and recreate the consul-client container so it will perform the auto_config request with the newly updated JSON web token (JWT).

$ docker rm consul-client --force && docker-compose up --detach consul-client

consul-client
Creating consul-client ... done

Validate success

Navigate to http://localhost:8500 on your browser to access the Consul UI. If not already logged in, login with the pre-configured ACL token e95b599e-166e-7d80-08ad-aee76e7ddf19.

Click on the "Nodes" option in the top navigation bar to go to the nodes page. Notice that the consul-client node is now present amongst the three consul-server# nodes. This indicates that consul-client has successfully joined the Consul datacenter using the auto_config method.

Image of Nodes page in Consul UI.

All future gossip encryption key, TLS certificate, and/or security setting changes will automatically be distributed to consul-client since it used auto_config to join the Consul datacenter.

Clean up your environment

To clean up your environment, execute the following command.

$ docker-compose down --rmi all

Stopping consul-server1 ... done
Stopping consul-client  ... done
Stopping consul-server3 ... done
Stopping consul-server2 ... done
Stopping vault-server   ... done
Removing consul-server1 ... done
Removing consul-client  ... done
Removing consul-server3 ... done
Removing consul-server2 ... done
Removing vault-server   ... done
Removing network docker-compose-datacenter_consul
Removing image consul:10.0.1

Next steps

In this tutorial, you learned to deploy and configure a secure local containerized Consul datacenter using Docker Compose. You learned how to use auto_config to send secure properties throughout your datacenter. You learned the value auto_config adds by automatically distributing all future gossip encryption key, TLS certificate, and/or other security setting changes across the datacenter. Finally, you learned how to clean up your environment.

You can continue learning how to deploy a Consul datacenter in production by completing the Deployment guide. The collection includes securing the datacenter with Access Control Lists, encryption, DNS configuration, and datacenter federation.

You can also extend your Consul skills by exploring the following tutorials:

For additional reference documentation on the official Docker images for Consul and Vault, refer to the following websites:

Previous

Autopilot

 Next

Observability with Docker Compose