Consul and Kubernetes deployment guide

This tutorial covers the necessary steps to install and configure a new Consul datacenter on Kubernetes, as defined in the Consul Reference Architecture tutorial. By the end of this tutorial, you will be able to:

• Identify installation prerequisites
• Customize the Helm chart to fit your environmental requirements
• Interact with your new Consul datacenter

Prerequisites

To complete this tutorial you will need:

• Access to a Kubernetes cluster
• kubectl
• helm v3.2.1+ or Consul-K8S-CLI

Configure Kubernetes permissions

Before deploying Consul, you will need to have kubectl credentials that allow you to create and modify policies, deploy services, create secrets, and create RBAC objects. You can find documentation for RBAC and service accounts for the following cloud providers using the links below.

• AKS
• EKS
• GCP

Configure Kubernetes secrets

You may wish to create Kubernetes secrets to store Consul data. Secrets can be referenced in the customized Helm chart values file.

Add an Enterprise license key (optional)

If you have purchased Enterprise Consul, you will need to add the enterprise license to the Kubernetes secretes engine. Also, you will need to specify Consul image using the Helm chart values file. The default image used by the official Consul helm chart is the open source version of Consul. The image name format will resemble this string: hashicorp/consul-enterprise:<latest version>-ent.

For Enterprise customers that have purchased a license, work with your Customer Success Manager (CSM) to get the appropriate license. Once you have obtained the license, you can use the following command to register it as a secret in Kubernetes.

$ kubectl create secret generic consul-license --from-literal="key=$(<YOUR LICENSE FROM CSM>)"

Configure Helm chart

Now that you have prepared your Kubernetes cluster, you can customize the Helm chart. First, you will need to add the HashiCorp Helm Chart repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

"hashicorp" has been added to your repositories

The Helm chart is configured with default values. You can override these values by creating your own config.yaml file and passing it to helm install with the -f flag, e.g. helm install consul hashicorp/consul -f config.yaml. Below we detail some parameters you can customize, and provide an example file. However, you should consider your particular production needs when configuring your chart.

Global key

The values under the global key will affect all the other parameters in the chart.

To enable all Consul components in the Helm chart, set enabled to true. This means servers, clients, Consul DNS, and the Consul UI will be installed with their defaults. You may also set the following global parameters based on your specific environment requirements or naming conventions.

• image is the name and tag of the Consul Docker image - override this if you want to run a specific version of Consul, or if you are using Consul Enterprise. Consult the Consul compatibility matrix before overriding the image setting to make sure the chart version you are using is compatible with the image you are specifying.
• datacenter the name of your Consul datacenter. The default value is dc1, but can be overridden if necessary.
• domain the domain Consul uses for DNS queries. The default value is consul but can be overridden if necessary.

For security, we recommend you set the global.acls.manageSystemACLs parameter to true. This will instruct Consul to initialize it's ACL system during chart installation.

Additionally, setting the global.gossipEncryption.autoGenerate parameter to true will configure gossip encryption automatically with a random key when Consul is installed.

We also recommend you read the Consul Helm chart documentation to review all the possible global parameters.

Consul UI

To enable the Consul web UI update the ui section to your values file and set enabled to true.

Consul servers

For production deployments, you should deploy 3 or 5 Consul servers for quorum and failure tolerance. For most deployments, 3 servers are adequate.

In the server section set replicas to 3. This will deploy three servers and cause Consul to wait to perform leader election until all three are healthy. The resources will depend on your environment; in the example at the end of the tutorial, the resources are set for a large environment.

Affinity

To ensure the Consul servers are placed on different Kubernetes nodes, you will need to configure affinity. Otherwise, the failure of one Kubernetes node could cause the loss of multiple Consul servers, and result in quorum loss. The default values.yaml has affinity configured correctly.

Enterprise license

If you have an Enterprise license, you should reference the Kubernetes secret in the enterpriseLicense parameter. Read the Consul Helm chart documentation to review the server parameters related to configuring an enterprise license.

Consul clients

Since a Consul client is deployed on every Kubernetes node, you do not need to specify the number of clients for your deployments. You will, however, need to specify resources. Since Consul clients are designed for horizontal scalability, the resources in the example at the end of this tutorial should be sufficient for most production scenarios.

It is worth mentioning that when configuring clients, you have the ability to override the default node reconnect timeout. By default, Consul will wait 72 hours before reaping unresponsive nodes. Since Consul on Kubernetes often runs on ephemeral infrastructure, such as spot instances, this default setting may not meet your needs. To override the default, you can set the client.extraConfig setting by passing a valid JSON object that includes an advertise_reconnect_timeout key set to the timeout interval that makes sense for your environment. As shown in this example.

client:
  extraConfig: |
    {"advertise_reconnect_timeout": "15m"}

To review all the possible client parameters, refer to the Consul Helm chart documentation.

Consul service mesh configuration

To enable Consul service mesh, set enabled under the connectInject key to true. In the connectInject section, you can also configure security features. Enabling the default parameter will result in the injector automatically injecting the envoy-sidecar sidecar proxy and consul-sidecar life-cycle sidecar into all pods. If you prefer to manually annotate which pods to inject, you should set connectInject.default to false.

Setting the aclBindingRuleSelector parameter to serviceaccount.name!=default ensures that new services do not all receive the same token if you are only using a default service account. This setting is only necessary if you have enabled ACLs in the global section.

Read more about the Connect Inject parameters.

Complete example

Your finished config.yaml file should resemble the following example. For more complete descriptions of all the available parameters, check the values.yaml file provided with the Helm chart and the reference documentation.

# Configure global settings in this section.
global:
  name: consul
  # Bootstrap ACLs within Consul. This is highly recommended.
  acls:
    manageSystemACLs: true
  # Gossip encryption
  gossipEncryption:
    autoGenerate: true
# Configure your Consul servers in this section.
server:
  # Specify three servers that wait until all are healthy to bootstrap the Consul cluster.
  replicas: 3
  # Specify the resources that servers request for placement. These values will serve a large environment.
  resources:
    requests:
      memory: '32Gi'
      cpu: '4'
      disk: '50Gi'
    limits:
      memory: '32Gi'
      cpu: '4'
      disk: '50Gi'
  # If using Enterprise, reference the Kubernetes secret that holds your license here
  enterpriseLicense:
    secretName: 'consul-license'
    secretKey: 'key'
# Configure Consul clients in this section
client:
  # Specify the resources that clients request for deployment.
  resources:
    requests:
      memory: '8Gi'
      cpu: '2'
      disk: '15Gi'
    limits:
      memory: '8Gi'
      cpu: '2'
      disk: '15Gi'
# Enable and configure the Consul UI.
ui:
  enabled: true
# Enable Consul connect pod injection
connectInject:
  enabled: true
  default: true
controller:
  enabled: true

Deploy Consul

Now that you have customized your config.yaml file, you can deploy Consul with Helm or the Consul K8S CLI. It is recommended to deploy Consul into its own dedicated namespace as shown below. This should only take a few minutes. The Consul pods should appear in the pod list immediately.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm install consul hashicorp/consul --values config.yaml --create-namespace --namespace consul --version "0.43.0"

$ brew tap hashicorp/tap

$ brew install hashicorp/tap/consul-k8s

$ consul-k8s install -config-file=config.yaml -set global.image=hashicorp/consul:1.12.0

To check the deployment process on the command line you can use kubectl get pods --namespace consul.

$ kubectl get pods --namespace consul

Next steps

In this tutorial, you configured Consul using the Helm chart for a production environment. This involved ensuring that your Kubernetes cluster had a properly distributed configuration for Consul servers, specifying enough resources for your agents, securing the datacenter with ACLs and gossip encryption, and enabling other Consul functionality including Connect service mesh and the Consul UI.

Now, you can interact with your Consul datacenter through the UI or CLI.

If you exposed the UI using a load balancer it will be available at the LoadBalancer Ingress IP address and Port that is output from the following command.

$ kubectl describe service consul-server --namespace consul

To access the Consul CLI, open a terminal session using the Kubernetes CLI.

$ kubectl exec --stdin --tty consul-server-0 --namespace consul -- /bin/sh

This will allow you to navigate the file system and run Consul CLI commands on the pod. For example you can view the Consul members.

$ consul members

When you have finished interacting with the pod, exit the shell.

$ exit

Next steps

In this tutorial, you installed and configured a new Consul datacenter on Kubernetes using the official Helm chart or Consul K8S CLI. To continue learning about Consul service discovery and service mesh, complete the Consul service mesh tutorials. For more information on securing and debugging Consul on Kubernetes review the Secure Consul and Registered Services on Kubernetes tutorial.