Deployment Guide

35min
|
Consul

This deployment guide covers the steps required to install and configure a secure HashiCorp Consul datacenter as defined in the Consul Reference Architecture.

This deployment guide is designed to work in combination with the Consul Reference Architecture. Although not a strict requirement to follow the Consul Reference Architecture, it is highly recommended that you are familiar with the overall architecture design; for example installing Consul server agents on multiple physical or virtual (with correct anti-affinity) hosts for high-availability.

To provide a highly-available single datacenter architecture, we recommend Consul server agents be deployed to more than one host, as shown in the Consul Reference Architecture.

Reference Diagram

These setup steps should be completed on all Consul hosts.

Install Consul

HashiCorp officially maintains and signs packages for the following Linux distributions.

INFO The steps provided from this tutorial require some packages to be available in your machine. Install them with the following command: apt-get install curl gnupg lsb-release

Add the HashiCorp GPG key.

$ curl --fail --silent --show-error --location https://apt.releases.hashicorp.com/gpg | \
      gpg --dearmor | \
      sudo dd of=/usr/share/keyrings/hashicorp-archive-keyring.gpg

Add the official HashiCorp Linux repository.

$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
 sudo tee -a /etc/apt/sources.list.d/hashicorp.list

Update.

$ sudo apt-get update

Install the latest version of Consul.

$ sudo apt-get install consul

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command sudo apt-cache policy consul.

$ sudo apt-get install consul=1.14.3-*

INFO Note that the version must be suffixed with -*. For details, refer to Linux Packaging Debian Revision Change.

Install the latest version of Consul Enterprise.

$ sudo apt-get install consul-enterprise

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command sudo apt-cache policy consul-enterprise.

$ sudo apt-get install consul-enterprise=1.14.3+ent-*

INFO Note that the version must be suffixed with -*. For details, refer to Linux Packaging Debian Revision Change.

Install yum-config-manager to manage your repositories.

$ sudo yum install -y yum-utils

Use yum-config-manager to add the official HashiCorp Linux repository.

$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

Install the latest version of Consul.

$ sudo yum -y install consul

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command yum --showduplicate list consul.

$ sudo yum install consul-1.14.3

Install the latest version of Consul Enterprise.

$ sudo yum -y install consul-enterprise

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command yum --showduplicate list consul-enterprise.

$ sudo yum install consul-enterprise-1.14.3+ent

Install dnf config-manager to manage your repositories.

$ sudo dnf install -y dnf-plugins-core

Use dnf config-manager to add the official HashiCorp Linux repository.

$ sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo

Install latest version.

$ sudo dnf install consul

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command sudo dnf --showduplicate list consul.

$ sudo dnf install consul-1.14.3

Install the latest version of Consul Enterprise.

$ sudo dnf install consul-enterprise

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command sudo dnf --showduplicate list consul-enterprise.

$ sudo dnf install consul-enterprise-1.14.3+ent

Install yum-config-manager to manage your repositories.

$ sudo yum install -y yum-utils

Use yum-config-manager to add the official HashiCorp Linux repository.

$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo

Install the latest version of Consul.

$ sudo yum -y install consul

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command yum --showduplicate list consul.

$ sudo yum install consul-1.14.3

Install the latest version of Consul Enterprise.

$ sudo yum -y install consul-enterprise

Alternatively, install a specific version. A list of released versions can be found at https://releases.hashicorp.com/consul/ or you can run the command yum --showduplicate list consul-enterprise.

$ sudo yum install consul-enterprise-1.14.3+ent

Tip

Now that you have added the HashiCorp repository, you can install Terraform, Vault, Nomad and Packer with the same command.

Homebrew is a free and open-source package management system for MacOS. Install the Consul formula from the terminal.

$ brew install consul

NOTE Homebrew and the Consul formula are NOT directly maintained by HashiCorp. The latest version of Consul is always available by manual installation.

Chocolatey is a free and open-source package management system for Windows. Install the Consul package from the command-line.

$ choco install consul

NOTE Chocolatey and the Consul package are NOT directly maintained by HashiCorp. The latest version of Consul is always available by manual installation.

INFO If you are installing with Chocolatey on a system with multiple NICs, it is safe to ignore the following error output since you will manually install Consul as a service in later steps: consul: Unexpected status SERVICE_STOPPED in response to START control.

Precompiled Consul binaries are available for download at https://releases.hashicorp.com/consul/. Instructions for running Consul Enterprise binaries are available for HashiCorp Consul customers.

You should perform checksum verification of the zip packages using the SHA256SUMS and SHA256SUMS.sig files available for the specific release version. HashiCorp provides a guide on checksum verification for precompiled binaries.

Export a couple of environment variables to specify the Consul download base URL and preferred Consul version. Then use curl to download the package.

$ export CONSUL_VERSION="1.14.3"

$ export CONSUL_URL="https://releases.hashicorp.com/consul"

$ curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip

Optionally perform the checksum verification.

$ curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS

$ curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS.sig

$ sha256sum --check --ignore-missing consul_${CONSUL_VERSION}_SHA256SUMS

Unzip the downloaded package and move the consul binary to /usr/bin/. Check consul is available on the system path.

$ unzip consul_${CONSUL_VERSION}_linux_amd64.zip

$ sudo chown root:root consul

$ sudo mv consul /usr/bin/

$ consul --version

The consul command features opt-in autocompletion for flags, subcommands, and arguments (where supported). Enable autocompletion.

$ consul -autocomplete-install

$ complete -C /usr/bin/consul consul

Create a unique, non-privileged system user to run Consul and create its data directory.

$ sudo useradd --system --home /etc/consul.d --shell /bin/false consul

$ sudo mkdir --parents /opt/consul

$ sudo chown --recursive consul:consul /opt/consul

In Powershell export a couple environment variables to specify the Consul download base URL and preferred Consul version.

$CONSUL_VERSION = '1.14.3'

$CONSUL_URL = 'https://releases.hashicorp.com/consul'

$CONSUL_DIR = "C:\Hashicorp\Consul"

Create the Consul directories

mkdir "$CONSUL_DIR"

mkdir "$CONSUL_DIR/data"

mkdir "$CONSUL_DIR/certs"

Change into the Consul directories and download the Consul package.

Set-Location "$CONSUL_DIR"

Invoke-WebRequest "${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_windows_amd64.zip" -Outfile consul_${CONSUL_VERSION}_windows_amd64.zip

Optionally perform the checksum verification.

Invoke-WebRequest "${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS" -Outfile consul_${CONSUL_VERSION}_SHA256SUMS

Invoke-WebRequest "${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS.sig" -Outfile consul_${CONSUL_VERSION}_SHA256SUMS.sig

get-content "${CONSUL_DIR}/*SHA256SUMS"| select-string (get-filehash -algorithm SHA256 "${CONSUL_DIR}/consul_${CONSUL_VERSION}_windows_amd64.zip").hash.toLower()

Expand out the zip file in your directory

Expand-Archive -Confirm:$false -Force:$true "${CONSUL_DIR}/consul_${CONSUL_VERSION}_windows_amd64.zip" "$CONSUL_DIR"

Add Consul directory to the system path (both current and future).

$env:path += ";${CONSUL_DIR}"

[Environment]::SetEnvironmentVariable("Path", [Environment]::GetEnvironmentVariable("Path", "Machine") + ";${CONSUL_DIR}", "Machine")

Verify the installation

After installing Consul, verify that the installation worked by opening a new terminal session and running the command consul.

$ consul

usage: consul [--version] [--help] <command> [<args>]

Available commands are:
    acl            Interact with Consul's ACLs
    agent          Runs a Consul agent
    catalog        Interact with the catalog
...

If you get an error that consul could not be found, your PATH environment variable was not set up properly. Make sure that your PATH variable contains the directory where you installed Consul.

Prepare the security credentials

In this section, you will create the encryption key and certificates needed to secure your Consul agents.

All commands in this section can be completed on any system with the Consul binary installed. The outputs and artifacts from these commands should be archived in a secure location for future reference.

Generate the gossip encryption key

Gossip is encrypted with a symmetric key, since gossip between nodes is done over UDP. All agents must have the same encryption key.

You can create the encryption key via the Consul CLI even though no Consul agents are running yet. Generate the encryption key.

$ consul keygen

The encryption key will be plain text output:

$ consul keygen
qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0=

NOTE You will need to add the newly generated key to the encrypt option in the configuration on all Consul clients and servers. Save your key in a safe location. You will need to reference the key throughout the installation.

For in-depth information about setting up gossip encryption, review the Secure Gossip Communication with Encryption tutorial.

Generate TLS certificates for RPC encryption

Consul can use TLS to verify the authenticity of servers and clients. To enable TLS, Consul requires that all agents have certificates signed by a single Certificate Authority (CA).

In this tutorial, you will use Consul to create a CA for your certificates. For in-depth information about setting up TLS certificates, review the Secure Consul Agent Communication with TLS Encryption tutorial.

Create the Certificate Authority

Start by creating the CA on your admin instance, using the Consul CLI.

$ consul tls ca create

==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

The next steps will use these files to create TLS certificates.

Create the certificates

Next, create a set of certificates for your Consul servers. You must select a name for your datacenter and domain so that the certificates are named properly.

Use the following command to create a certificate for each server. The file name increments automatically.

$ consul tls cert create -server -dc <dc_name> -domain <domain>

Below is an example for datacenter "dc1" in "consul" domain. These are the defaults but for this example they are set explicitly.

$ consul tls cert create -server -dc dc1 -domain consul

==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

A federated Consul environment requires the server certificate to include the names of all Consul datacenters that are within the federated environment.

The -additional-dnsname flag allows you to provide an additional DNS name for Subject Alternative Names. localhost is always included. This flag may be provided multiple times.

$ consul tls cert create -server -dc <dc_name> -domain <domain> -additional-dnsname=<secondary_consul_server_name>

Below is an example for a federated environment that contains two consul datacenters dc1 and dc2 in the consul domain.

$ consul tls cert create -server -dc dc1 -domain consul -additional-dnsname=*.dc2.consul

==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

Warning

If you plan to use a custom domain for resolving DNS queries, you must pass the same domain using the -domain flag when generating TLS certificates.

Note

Check the consul TLS cert documentation to learn more about TLS cert creation options.

Consul client agents can be configured in two ways: via auto-encrypt or manual configuration.

The recommended approach is to use the auto encryption mechanism provided by Consul that automatically generates client certificates using the previously created CA certificate consul-agent-ca.pem to enable TLS.

You will learn how to enable auto-encrypt in Configure Consul agents.

The manual configuration generates a certificate/key pair for each Consul client agent.

Use the following command with the -client flag to create client certificates. The file name increments automatically.

$ consul tls cert create -client -dc <dc_name>

Below is an example for datacenter "dc1":

$ consul tls cert create -client -dc dc1

==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem

Distribute the certificates to agents

You must distribute the CA certificate, consul-agent-ca.pem, to each of the Consul agents as well as the agent-specific certificate and private key:

If a server agent, distribute the CA file, server certificate, and server private key.
If a client agent using auto-encrypt, distribute the CA file only.
If a client agent using manual configuration, distribute the CA file, client certificate, and client private key.

Below is an SCP example which will send the CA certificate, agent certificate and private key to the IP address you specify, and put it into the /etc/consul.d/certs directory.

$ scp consul-agent-ca.pem <dc-name>-<server/client>-consul-<cert-number>.pem <dc-name>-<server/client>-consul-<cert-number>-key.pem <USER>@<PUBLIC_IP>:/etc/consul.d/certs

NOTE Remember the file path to the CA certificate. It is recommended to store them in a directory like /etc/consul.d/certs or C:\Hashicorp\consul\certs. You will need to include the entire file path in the server configuration file and when using the API or CLI.

Configure Consul agents

Consul uses documented reasonable defaults so only non-default values must be set in the configuration file. Configuration can be read from multiple files and is loaded in lexical order. Check the full description for more information about configuration loading and merge semantics.

Consul server agents typically require a superset of configuration required by Consul client agents. You will specify common configuration used by all Consul agents in consul.hcl and server specific configuration in server.hcl.

Create a configuration file at /etc/consul.d/consul.hcl:

$ sudo mkdir --parents /etc/consul.d

$ sudo touch /etc/consul.d/consul.hcl

$ sudo chown --recursive consul:consul /etc/consul.d

$ sudo chmod 640 /etc/consul.d/consul.hcl

Add this configuration to the consul.hcl configuration file:

NOTE Replace the datacenter parameter value with the identifier you will use for the datacenter this Consul agent is deployed in. Replace the encrypt parameter value with the previously generated output from Generate the gossip encryption key.

datacenter = "dc1"
data_dir = "/opt/consul"
encrypt = "qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0="

datacenter - The datacenter in which the agent is running.
data_dir - The data directory for the agent to store state.
encrypt - Specifies the gossip encryption key to use for Consul network traffic.

TLS configuration

To secure RPC communications for Consul clients consul provides two different approaches, the auto encryption mechanism and the manual configuration.

tls {
   defaults {
      ca_file = "<Consul configuration directory>/certs/consul-agent-ca.pem"
      cert_file = "<Consul configuration directory>/certs/dc1-server-consul-0.pem"
      key_file = "<Consul configuration directory>/certs/dc1-server-consul-0-key.pem"

      verify_incoming = true
      verify_outgoing = true
   }
   internal_rpc {
      verify_server_hostname = true
   }
}

auto_encrypt {
  allow_tls = true
}

tls {
   defaults {
      ca_file = "<Consul configuration directory>/certs/consul-agent-ca.pem"

      verify_incoming = true
      verify_outgoing = true
   }
   internal_rpc {
      verify_server_hostname = true
   }
}

auto_encrypt {
  tls = true
}

tls {
   defaults {
      ca_file = "<Consul configuration directory>/certs/consul-agent-ca.pem"
      cert_file = "<Consul configuration directory>/certs/dc1-server-consul-0.pem"
      key_file = "<Consul configuration directory>/certs/dc1-server-consul-0-key.pem"

      verify_incoming = true
      verify_outgoing = true
   }
   internal_rpc {
      verify_server_hostname = true
   }
}

tls {
   defaults {
      ca_file = "<Consul configuration directory>/certs/consul-agent-ca.pem"
      cert_file = "<Consul configuration directory>/certs/dc1-client-consul-0.pem"
      key_file = "<Consul configuration directory>/certs/dc1-client-consul-0-key.pem"

      verify_incoming = true
      verify_outgoing = true
   }
   internal_rpc {
      verify_server_hostname = true
   }
}

tls.defaults.ca_file - Specifies the path to the CA public certificate file.
tls.defaults.cert_file - Specifies the path to the agents public certificate file.
tls.defaults.key_file - Specifies the path to the agents certificate private key file.
tls.defaults.verify_incoming - If set to true, Consul requires all incoming connections to use TLS.
tls.defaults.verify_outgoing - If set to true, Consul requires that all outgoing connections from this agent use TLS.
tls.internal_rpc.verify_server_hostname - If set to true, Consul verifies for all outgoing TLS connections that the TLS certificate presented by the servers matches server.<datacenter>.<domain> hostname.

If using a custom domain for DNS queries, specify the domain field as well. This should match the -domain flags that were used when generating TLS credentials in Generate TLS certificates for RPC encryption:

domain = "example.com"

Datacenter auto-join

The retry_join parameter allows you to configure all Consul agents to automatically form a datacenter using a common Consul server accessed via DNS address, IP address or using Cloud Auto-join. This removes the need to manually join the Consul datacenter nodes together.

Add the retry_join parameter to the consul.hcl configuration file:

NOTE Replace the retry_join parameter value with the correct DNS address, IP address, Loopback address or cloud auto-join configuration for your environment.

Warning

The retry_join parameter is required for the systemd process to complete successfully and send its notify signal on LAN join.

retry_join = ["172.16.0.11"]

retry_join - Address of another agent to join upon starting up.

Enable Consul ACLs

Consul uses Access Control Lists (ACLs) to secure the UI, API, CLI, and Consul catalog including service and agent registration. When securing your datacenter you should configure the ACLs first.

The Secure Consul with Access Control Lists (ACLs) tutorial provides instructions on configuring and enabling ACLs on new agents.

Add the ACL configuration to the consul.hcl configuration file and choose a default policy of "allow" (allow all traffic unless explicitly denied) or "deny" (deny all traffic unless explicitly allowed). Default policy of "deny" is recommended for production.

acl {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

acl - ACL stanza
enabled - Enables ACLs
default_policy - The default policy controls the behavior of a token when there is no matching rule.
enable_token_persistence - When true tokens set using the API will be persisted to disk and reloaded when an agent restarts.

Telemetry stanza

The telemetry stanza specifies various configurations for Consul to publish metrics to upstream systems.

If you decide to configure Consul to publish telemetry data, you should complete the Monitor Consul Datacenter Health tutorial.

Audit

Enterprise Only

The audit logging functionality demonstrated here requires HashiCorp Cloud Platform (HCP) or self-managed Consul Enterprise. If you've purchased or wish to try out Consul Enterprise, refer to how to access Consul Enterprise.

Warning

ACLs must be enabled to use this feature

The audit stanza, added in Consul 1.8, allow users to enable auditing and configure a sink and filters for their audit logs.

Add the audit configuration to the consul.hcl configuration file:

audit {
  enabled = true
  sink "My sink" {
    type   = "file"
    format = "json"
    path   = "data/audit/audit.json"
    delivery_guarantee = "best-effort"
    rotate_duration = "24h"
    rotate_max_files = 15
    rotate_bytes = 25165824
  }
}

Configuration options for the audit stanza can be found here.

Server specific configuration

Create a configuration file at /etc/consul.d/server.hcl:

$ sudo touch /etc/consul.d/server.hcl

$ sudo chown --recursive consul:consul /etc/consul.d

$ sudo chmod 640 /etc/consul.d/server.hcl

Add this configuration to the server.hcl configuration file.

NOTE Replace the bootstrap_expect value with the number of Consul servers you will use; three or five is recommended.

server = true
bootstrap_expect = 3

server - This flag is used to control if an agent is in server or client mode.
bootstrap_expect - This flag provides the number of expected servers in the datacenter. Either this value should not be provided or the value should be consistent across all servers in the datacenter.

Create a configuration file at C:\Hashicorp\Consul\server.hcl.

New-Item -ItemType "file" -Path "C:\Hashicorp\Consul\server.hcl"

Add this configuration to the server.hcl configuration file:

NOTE Replace the bootstrap_expect value with the number of Consul servers you will use; three or five is recommended.

@"
server = true
bootstrap_expect = 3
"@ | Out-File -Encoding ASCII -FilePath "${CONSUL_DIR}/server.hcl"

server - This flag is used to control if an agent is in server or client mode.
bootstrap_expect - This flag provides the number of expected servers in the datacenter. Either this value should not be provided or the value should be consistent across all servers in the datacenter.

Consul addresses

By default, Consul will bind on all available addresses on the machine and expose client interfaces only on localhost/127.0.0.1.

Internal communications address binding

Consul agent will bind to all addresses on the local machine and will advertise the private IPv4 address to the rest of the datacenter.

There are some cases when you want to specify the bind address for your Consul server:

the Consul node has multiple private IP addresses available
the Consul node has only public IP addresses available

If there are multiple private IPv4 addresses available, Consul will exit with an error at startup:

Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.

To prevent this from happening it is possible to define on which address Consul will bind using the bind_addr parameter.

bind_addr = "172.16.0.10"

If there are only public IP addresses available, Consul will exit with an error at startup:

"Error starting agent: Failed to get advertise address: No private IP address found".

To prevent this from happening it is possible to define on which address Consul will bind using the bind_addr parameter.

bind_addr = "<One of Consul's public IPs>"

bind_addr - The address that should be bound to for internal cluster communications. By default, this is "0.0.0.0", meaning Consul will bind to all addresses on the local machine.

Client address binding

Consul agents designate an interface to which Consul services will bind, including the HTTP and DNS servers.

When the value for client_addr is undefined, it defaults to "127.0.0.1", allowing only loopback connections.

Optionally, you can specify a bind IP address in your Consul server.hcl configuration file.

client_addr = "0.0.0.0"

client_addr - The address that Consul will bind client interfaces to, including the HTTP and DNS servers. By default. By default it is "127.0.0.1", allowing only loopback connections.

Performance stanza

The performance stanza allows tuning the performance of different subsystems in Consul.

Add the performance configuration to the server.hcl configuration file:

performance {
  raft_multiplier = 1
}

raft_multiplier - An integer multiplier used by Consul servers to scale key Raft timing parameters. Setting this to a value of 1 will configure Raft to its highest-performance mode, equivalent to the default timing of Consul prior to 0.7, and is recommended for production Consul servers.

For more information on Raft tuning and the raft_multiplier setting, check the server performance documentation.

Consul service mesh

Consul service mesh provides service-to-service connection authorization and encryption using mutual Transport Layer Security (TLS). Applications can use sidecar proxies in a service mesh configuration to establish TLS connections for inbound and outbound connections without being aware of Consul service mesh.

Consul service mesh uses the registered service identity (rather than IP addresses) to enforce access control with intentions. Add the following configuration to the server.hcl configuration file.

connect {
  enabled = true
}

addresses {
  grpc = "127.0.0.1"
}

ports {
  grpc  = 8502
}

connect.enabled - Controls whether Connect features are enabled on this agent. Should be enabled on all servers in the cluster in order for Connect to function properly.
addresses.grpc - The address that Consul will bind gRPC API to. Defaults to client_addr but it might be sensitive to have it on localhost/127.0.0.1 for security reasons.
ports.grpc - The gRPC API port. We recommend using 8502 for grpc by convention as some tooling will work automatically with this. Currently gRPC is only used to expose Envoy xDS API to Envoy proxies.

Note

By default, Consul service mesh uses an embedded CA to generate and sign certificates for services. It is possible to configure Consul to use a different CA for it.

Consul UI

Consul features a web-based user interface, allowing you to get an overview of all services, nodes, intentions and more using a graphical user interface, rather than the CLI or API.

NOTE You should consider running the Consul UI on selected Consul hosts rather than all hosts.

INFO By default the UI binds to the client_addr which is 127.0.0.1.

Optionally, add the UI configuration to your server.hcl configuration file to enable the Consul UI:

ui_config {
  enabled = true
}

ui - Enables the built-in web UI.

Client specific configuration

Consul client agents typically require a subset of configuration required by Consul server agents. All Consul clients can use the consul.hcl file created earlier in the configuring the Consul agents section. If you have added host-specific configuration such as identifiers, you will need to set these individually.

Apply Enterprise license

In order to run Consul Enterprise you need a valid Enterprise license.

If you are using Consul Enterprise version 1.10.0+, check the Installing HashiCorp Enterprise License tutorial for guidance on how to apply an enterprise license.

You will be able to temporarily run Consul Enterprise without a license when using the Enterprise version of Consul binaries < 1.10.0. If you have purchased an Enterprise license, work with your Customer Success Manager (CSM) or email softwaredelivery@hashicorp.com to get the appropriate license. The following steps describe how to apply the Enterprise license.

Verify that you are using an Enterprise binary.
The license commands are only available for enterprise versions of the binaries. Make sure enterprise versions of the binaries have been installed and are running on the servers.
```
consul version
```
Consul Enterprise versions end with +ent.
```
Consul v1.7.2+ent
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```
NOTE You will have 6 hours to apply commands once enterprise binaries prio to 1.10.0 are up and running. Consul will shut down when the time limit expires. Consul Enterprise version >1.10.0 require a license key available during the start-up process, otherwise the server instance will not start.

Run the following command on the Consul leader.

$ consul license put "<paste license text here>"

Once the command runs successfully, the output will indicate that the license is valid.

License is valid
License ID: a763ad9f-ecfa-948c-7355-5a502aeac721
Customer ID: 7e6ac3e0-f44b-23c3-c486-3264eabc4326
Expires At: 2018-06-30 06:59:59.999 +0000 UTC
Datacenter: *
Package: cHJlbWl1bQ==
Licensed Features:
    Automated Backups
    Automated Upgrades
    Enhanced Read Scalability
    Network Segments
    Redundancy Zone
    Advanced Network Federation

The license will be replicated to other the Consul servers from the leader.

Configure the Consul process

If you installed Consul using one of the supported package managers on Linux here, you can skip the following step as systemd will be configured for you.

Configure systemd

Systemd uses documented reasonable defaults so only non-default values must be set in the configuration file.

Create a Consul service file at /etc/systemd/system/consul.service.

$ sudo touch /etc/systemd/system/consul.service

Add this configuration to the Consul service file.

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
EnvironmentFile=-/etc/consul.d/consul.env
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

The following parameters are set for the [Unit] stanza:

Description - Free-form string describing the consul service
Documentation - Link to the consul documentation
Requires - Configure a requirement dependency on the network service
After - Configure an ordering dependency on the network service being started before the consul service
ConditionFileNotEmpty - Check for a non-zero sized configuration file before consul is started

The following parameters are set for the [Service] stanza:

EnvironmentFile - Sets environment variables for Consul
User, Group - Run Consul as the consul user
ExecStart - Start Consul with the agent argument and path to the configuration file
ExecReload - Send Consul a reload signal to trigger a configuration reload in Consul
ExecStop - Stop Consul with the leave argument to allow for a graceful leave from the datacenter
KillMode - Treat consul as a single process
Restart - Restart consul unless it returned a clean exit code
LimitNOFILE - Set an increased limit for file descriptors

The following parameters are set for the [Install] stanza:

WantedBy - Creates a weak dependency on Consul being started by the multi-user run level

Configure Consul service

Generate a random password for your Consul service account.

$password = (-join ((0x30.. 0x39) + ( 0x41.. 0x5A) + ( 0x61.. 0x7A) | Get-Random -Count 16 | % {[char]$_}))

$securePassword = (ConvertTo-SecureString -AsPlainText -Force -String $password)

Create a unique, non-privileged system user to run consul.

New-LocalUser "consul" -FullName "Consul User" -Description "Consul Service Account" -Password $securePassword

$Credential = New-Object -TypeName System.Management.Automation.PSCredential(".\consul", $securePassword)

NOTE Local GPO is not able to be edited natively via PowerShell, set account logon rights manually.

Create the Consul service

Option 1: Local system user

New-Service -Name Consul -BinaryPathName "${CONSUL_DIR}/consul.exe agent -config-dir=${CONSUL_DIR}"  -DisplayName Consul -Description "Hashicorp Consul Service https://consul.io" -StartupType "Automatic"

Option 2: Consul User

NOTE Add 'Consul' User to the Local or AD GPO Policy to LogOnAsAservice.

New-Service -Name Consul -BinaryPathName "${CONSUL_DIR}/consul.exe agent -config-dir=${CONSUL_DIR}" -DisplayName Consul -Description "Hashicorp Consul Service https://consul.io" -StartupType "Automatic" -Credential $Credential

Start the Consul service

Check that your configuration file is valid, with the Consul CLI validate command.

$ sudo consul validate /etc/consul.d/
...
Configuration is valid!

Enable and start Consul using the systemctl command responsible for controlling systemd managed services. Check the status of the consul service using systemctl.

$ sudo systemctl enable consul

$ sudo systemctl start consul

$ sudo systemctl status consul

The output should indicate the service is running.

$ sudo systemctl status consul
● consul.service - Consul
   Loaded: loaded (/usr/lib/systemd/system/consul.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-09-06 18:44:55 UTC; 4s ago
     Docs: https://www.consul.io/docs/
 Main PID: 13244 (consul)
    Tasks: 10
   Memory: 10.2M
      CPU: 86ms
   CGroup: /system.slice/consul.service
           └─13244 /usr/bin/consul agent -config-dir=/etc/consul.d

Sep 06 18:44:56 consul201-bison.node.consul consul[13244]:     2019/09/06 18:44:56 [INFO] a
Sep 06 18:44:56 consul201-bison.node.consul consul[13244]:     2019/09/06 18:44:56 [INFO] a
Sep 06 18:44:56 consul201-bison.node.consul consul[13244]: ==> Consul agent running!

Setup Consul environment variables

Notice that since TLS encryption is enabled, you will now need to use a certificate to communicate with Consul and to complete all other tasks.

To make it easier to use the CLI for the remainder of the setup, and to complete the ACL bootstrapping process, set the following environment variables for all your agents.

If using auto-encrypt, the CONSUL_CLIENT_CERT and CONSUL_CLIENT_KEY environment variables are not required for client agents.

export CONSUL_CACERT=/etc/consul.d/consul-agent-ca.pem
export CONSUL_CLIENT_CERT=/etc/consul.d/<dc-name>-<server/client>-consul-<cert-number>.pem
export CONSUL_CLIENT_KEY=/etc/consul.d/<dc-name>-<server/client>-consul-<cert-number>-key.pem

$env:CONSUL_CACERT="C:\Hashicorp\Consul\certs\consul-agent-ca.pem"
$env:CONSUL_CLIENT_KEY="C:\Hashicorp\Consul\<dc-name>-<server/client>-consul-<cert-number>.pem"
$env:CONSUL_CLIENT_CERT="C:\Hashicorp\Consul\<dc-name>-<server/client>-consul-<cert-number>-key.pem"

Bootstrap the ACL system

The Secure Consul with Access Control Lists (ACLs) tutorial provides instructions on configuring and enabling ACLs on new agents. The below guidance assumes Consul has already been started without ACLs configured.

Create the initial bootstrap token

Working from one agent generate the Consul bootstrap token, which has unrestricted privileges.
```
$ consul acl bootstrap
```
This will return the Consul bootstrap token. You will need the SecretID for all subsequent Consul API requests (including CLI and UI). Ensure that you save the SecretID.
Set CONSUL_MGMT_TOKEN env variable. This is simply a convenience variable for the following configuration steps.
```
$ export CONSUL_MGMT_TOKEN="<Token SecretID from previous step>"
```
Create a directory to store our policy files.
```
mkdir policies
```

In the policies directory, create a node policy file (node-policy.hcl) with write access for nodes related actions and read access for service related actions.

agent_prefix "" {
  policy = "write"
}
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
session_prefix "" {
  policy = "read"
}

Generate the Consul node ACL policy with the newly created policy file.

$ consul acl policy create \
  -token=${CONSUL_MGMT_TOKEN} \
  -name node-policy \
  -rules @policies/node-policy.hcl

Create the node token with the newly created policy.

$ consul acl token create \
  -token=${CONSUL_MGMT_TOKEN} \
  -description "node token" \
  -policy-name node-policy

On all Consul Servers add the node token.

$ consul acl set-agent-token \
  -token=${CONSUL_MGMT_TOKEN} \
  agent "<Node Token SecretID>"

ACL token "agent" set successfully

Working from one agent generate the Consul bootstrap token, which has unrestricted privileges.
```
consul acl bootstrap
```
This will return the Consul bootstrap token. You will need the SecretID for all subsequent Consul API requests (including CLI and UI). Ensure that you save the SecretID.

Set the CONSUL_MGMT_TOKEN environment variable so you can create policies

$env:CONSUL_MGMT_TOKEN = `
foreach ( $token in $acl_tokens){
  if ($token.Split(':')[0] -match "SecretID") {($token.Split(':')[1]).trim();}
}

Create a directory to store our policy files.
```
mkdir "$CONSUL_DIR/policies"
```

Create a node policy file (node-policy.hcl) with write access for nodes related actions and read access for service related actions.

@'
agent_prefix "" {
  policy = "write"
}
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
session_prefix "" {
  policy = "read"
}
'@  | Out-File -Encoding ASCII -FilePath "${CONSUL_DIR}/policies/node-policy.hcl"

Generate the Consul node ACL policy with the newly created policy file.

consul acl policy create -token "${env:CONSUL_MGMT_TOKEN}" -name "node-policy" -rules "@${CONSUL_DIR}/policies/node-policy.hcl"

Create the node token with the newly created policy.

$acl_tokens = (consul acl token create -token "${env:CONSUL_MGMT_TOKEN}" -description "node token" -policy-name "node-policy")

Set the CONSUL_AGENT_TOKEN environment variable

$env:CONSUL_AGENT_TOKEN  = `
foreach ( $token in $acl_tokens){
  if ($token.Split(':')[0] -match "SecretID") {($token.Split(':')[1]).trim();}
}

On all Consul Servers add the node token.

consul acl set-agent-token -token "${env:CONSUL_MGMT_TOKEN}" agent "$env:CONSUL_AGENT_TOKEN"

NOTE For in-depth information about setting up tokens and ACLs for services, review the Secure Consul with Access Control Lists (ACLs) tutorial.

Next steps

In this tutorial you configured servers and clients in accordance to the reference architecture. This is the first step in deploying your first datacenter. In the next tutorial, you will learn how to configure backups to ensure the datacenter state is protected against possible failures.

To increase security for your datacenter, we recommend completing the Secure Consul with Access Control Lists (ACLs), Secure Gossip Communication with Encryption, and Secure Consul Agent Communication with TLS Encryption.

Consul security considerations

Production readiness checklist