Securely connect your services with Consul service mesh

In the previous tutorial, you deployed Consul client agents and registered services to your Consul catalog.

In this tutorial, you will introduce zero trust security in your network by implementing Consul service mesh. This will enable secure service-to-service communication and allow you to leverage Consul's full suite of features.

In order to do this, you will edit the service definitions on your Consul clients, launch Envoy sidecar proxies, and create service intentions to allow traffic across your services in your network.

In this tutorial, you will use a test environment to:

• Review and create intentions to manage traffic permissions
• Modify Consul services' configuration for Consul service mesh
• Start Envoy sidecar proxy for each service in the mesh
• Restart the services to listen on localhost interface

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a Consul datacenter with one server and four clients running on VMs. The services connect directly to each other using the VMs address and access every service in the network.

By the end of this tutorial, you will have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service. The services will be configured so they cannot be reachable, unless explicitly allowed through Consul service intentions.

Prerequisites

This tutorial uses an interactive lab to guide you through how to setting up service mesh on your VM workloads. The lab environment includes all required binaries and sample configurations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Verify that the VM you want to deploy the Consul server on has the Consul binary.

$ consul version
Consul v1.16.0
Revision 192df66a
Build Date 2023-06-26T20:07:11Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Verify Envoy installation on the VMs

In the interactive lab environment, click on each tab associated with the client nodes (NGINX, Frontend, API, and Database) and verify Envoy is installed.

$ envoy --version
envoy  version: cfa32deca25ac57c2bbecdad72807a9b13493fc1/1.26.4/Clean/RELEASE/BoringSSL

Configure environment

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The interactive lab environment includes these scripts. In Bastion Host, list the files in the ops directory.

$ tree ./ops
./ops
`-- scenarios
    `-- 99_supporting_scripts
        |-- generate_consul_client_config.sh
        |-- generate_consul_monitoring_config.sh
        |-- generate_consul_server_config.sh
        |-- generate_consul_server_tokens.sh
        |-- generate_consul_service_config.sh
        `-- generate_consul_service_intentions.sh

3 directories, 6 files

The scripts rely on default parameters to generate the configuration files. Set the following default values. Ensure you have permission to write in the specified paths.

The lab contains two environment files that will help you configure the terminal and generate the required configuration files.

$ ls -1 assets/scenario/*.env
assets/scenario/env-consul.env
assets/scenario/env-scenario.env

Import the two files in your environment.

$ source assets/scenario/env-scenario.env; \
source assets/scenario/env-consul.env

The script creates all the files in a destination folder. Export the path where you wish to create the configuration files for the scenario.

$ export OUTPUT_FOLDER="./assets/scenario/conf/";

Make sure the folder exists.

$ mkdir -p ${OUTPUT_FOLDER}

Verify your Consul CLI can interact with your Consul server.

$ consul members
Node                Address           Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0     10.2.70.61:8301   alive   server  1.16.0  2         dc1  default    <all>
hashicups-api       10.2.33.98:8301   alive   client  1.16.0  2         dc1  default    <default>
hashicups-db        10.2.48.253:8301  alive   client  1.16.0  2         dc1  default    <default>
hashicups-frontend  10.2.12.156:8301  alive   client  1.16.0  2         dc1  default    <default>
hashicups-nginx     10.2.22.45:8301   alive   client  1.16.0  2         dc1  default    <default>

Review and create intentions

The initial Consul configuration denies all service connections by default. We recommend this setting in production environments to follow the "least-privilege" principle, by restricting all network access unless explicitly defined.

Intentions let you allow and restrict access between services. Intentions are destination-orientated — this means you create the intentions for the destination, then define which services can access it.

The following intentions are required for HashiCups:

1. The db service needs to be reached by the api service.
2. The api service needs to be reached by the nginx services.
3. The frontend service needs to be reached by the nginx service.

Use the provided script to generate service intentions.

$ bash ops/scenarios/99_supporting_scripts/generate_consul_service_intentions.sh
Parameter Check
Create global proxy configuration
Create intention configuration files

Check the files generated by the script.

$ tree ${OUTPUT_FOLDER}/global
./assets/scenario/conf//global
├── config-global-proxy-default.hcl
├── config-global-proxy-default.json
├── intention-allow-all.hcl
├── intention-allow-all.json
├── intention-api.hcl
├── intention-api.json
├── intention-db.hcl
├── intention-db.json
├── intention-frontend.hcl
└── intention-frontend.json

0 directories, 10 files

Finally apply the intentions to your Consul datacenter.

$ consul config write ${OUTPUT_FOLDER}global/intention-db.hcl  
Config entry written: service-intentions/hashicups-db

$ consul config write ${OUTPUT_FOLDER}global/intention-api.hcl 
Config entry written: service-intentions/hashicups-api

$ consul config write ${OUTPUT_FOLDER}global/intention-frontend.hcl 
Config entry written: service-intentions/hashicups-frontend

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443:consul-server-0:8443 \
  --cacert ${CONSUL_CACERT} \
  --data @${OUTPUT_FOLDER}global/intention-db.json \
  --request PUT \
  https://server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443/v1/config

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443:consul-server-0:8443 \
  --cacert ${CONSUL_CACERT} \
  --data @${OUTPUT_FOLDER}global/intention-api.json \
  --request PUT \
  https://server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443/v1/config

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443:consul-server-0:8443 \
  --cacert ${CONSUL_CACERT} \
  --data @${OUTPUT_FOLDER}global/intention-frontend.json \
  --request PUT \
  https://server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443/v1/config

Apply new Consul client configuration

Before you can apply the new service configuration, you must copy the definition files into each Consul client VMs.

First, configure the Consul configuration directory.

$ export CONSUL_REMOTE_CONFIG_DIR=/etc/consul.d/

Then, copy the configuration into the client VMs.

$ scp -O -i certs/id_rsa ${OUTPUT_FOLDER}/hashicups-db/svc/service_mesh/* hashicups-db:${CONSUL_REMOTE_CONFIG_DIR}; \
  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}/hashicups-api/svc/service_mesh/* hashicups-api:${CONSUL_REMOTE_CONFIG_DIR}; \
  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}/hashicups-frontend/svc/service_mesh/* hashicups-frontend:${CONSUL_REMOTE_CONFIG_DIR}; \
  scp -O -i certs/id_rsa ${OUTPUT_FOLDER}/hashicups-nginx/svc/service_mesh/* hashicups-nginx:${CONSUL_REMOTE_CONFIG_DIR}

Start sidecar proxies on client VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and restart the Consul agent.

Start sidecar proxy for Database

Select the tab that corresponds with the service — in this case, Database.

Define the Consul configuration directory.

$ export CONSUL_CONFIG_DIR=/etc/consul.d/

Setup a valid token to interact with Consul agent.

$ export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Reload consul configuration.

$ consul reload
Configuration reload triggered

Finally, start the Envoy sidecar proxy for the service.

$ /usr/bin/consul connect envoy \
    -token=${CONSUL_HTTP_TOKEN} \
    -envoy-binary /usr/bin/envoy \
    -sidecar-for hashicups-db-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Start sidecar proxy for API

Select the tab that corresponds with the service — in this case, API.

Define the Consul configuration directory and setup a valid token to interact with Consul agent.

$ export CONSUL_CONFIG_DIR=/etc/consul.d/ && \
  export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Reload consul configuration.

$ consul reload
Configuration reload triggered

Finally, start the Envoy sidecar proxy for the service.

$ /usr/bin/consul connect envoy \
    -token=${CONSUL_HTTP_TOKEN} \
    -envoy-binary /usr/bin/envoy \
    -sidecar-for hashicups-api-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Start sidecar proxy for Frontend

Select the tab that corresponds with the service — in this case, Frontend.

Define the Consul configuration directory and setup a valid token to interact with Consul agent.

$ export CONSUL_CONFIG_DIR=/etc/consul.d/ && \
  export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Reload consul configuration.

$ consul reload
Configuration reload triggered

Finally, start the Envoy sidecar proxy for the service.

$ /usr/bin/consul connect envoy \
    -token=${CONSUL_HTTP_TOKEN} \
    -envoy-binary /usr/bin/envoy \
    -sidecar-for hashicups-frontend-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Start sidecar proxy for NGINX

Select the tab that corresponds with the service — in this case, NGINX.

Define the Consul configuration directory and setup a valid token to interact with Consul agent.

$ export CONSUL_CONFIG_DIR=/etc/consul.d/ && \
  export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Reload consul configuration.

$ consul reload
Configuration reload triggered

Finally, start the Envoy sidecar proxy for the service.

$ /usr/bin/consul connect envoy \
    -token=${CONSUL_HTTP_TOKEN} \
    -envoy-binary /usr/bin/envoy \
    -sidecar-for hashicups-nginx-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Restart services to listen on localhost

Now that the service configuration is applied, intentions are applied, and Envoy sidecars are started for each service, all the components for the Consul service mesh are in place. The Consul sidecar proxies will route the services' traffic to the target destination.

Since traffic is flowing through the sidecar proxies, you no longer need to expose your services externally. As a result, reconfigure them to listen on the loopback interface only to improve overall security.

Select the Bastion Host tab.

Reload the services to operate on the localhost interface.

$ ssh -i certs/id_rsa hashicups-db "bash -c 'bash ./start_service.sh local'"; \
  ssh -i certs/id_rsa hashicups-api "bash -c 'bash ./start_service.sh local'"; \
  ssh -i certs/id_rsa hashicups-frontend "bash -c 'bash ./start_service.sh local'"; \
  ssh -i certs/id_rsa hashicups-nginx "bash -c 'bash ./start_service.sh mesh'"

This tutorial still configures the NGINX service to listen on the VM's IP so you can still access it remotely. For production, we recommend using an ingress gateway to manage access to the service.

Click on the HashiCups tab.

Confirm that HashiCups still works despite its services being configured to communicate on localhost. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.

Next steps

In this tutorial, you learned how to migrate your Consul services from service discovery to Consul service mesh by updating each service's definitions, starting Envoy sidecar proxies for each service, and updating the services' dependencies to bind to localhost.

In the process, you integrated Zero Trust Security in your network and learned how to define explicitly define service-to-service permissions using intentions.

At this point, the NGINX service used to expose the application externally is still accessible over an insecure connection. While it is possible to configure it for secure traffic, using TLS, Consul offers an integrated solution for this.

In the next tutorial, you will learn how to add a Consul API Gateway to your service mesh and secure external network access to applications and services running in your Consul service mesh.

For more information about the topics covered in this tutorial, refer to the following resources:

• Consul Service Mesh
• Service Mesh Intentions
• Service Mesh Certificate Authority Overview