Connect services across datacenters with mesh gateways

Consul is a service mesh control plane that helps you to observe and secure service-to-service communication.

Consul service mesh secures network traffic between services using mutual TLS. "Intentions" allow you to control communication between individual services by allowing or denying communication between them. These security features work across datacenters, with automatic replication of intentions.

Mesh gateways, which connect services in different datacenters, take advantage of Server Name Indication (SNI) – a TLS extension – to inspect the destination of a connection and route to a remote datacenter without decrypting the payload. Using mesh gateways for inter-datacenter communication eliminates the requirement for sidecar proxies to have routable IP addresses across datacenters, freeing operators from complex IP address management to avoid IP overlap between datacenters.

When using Kubernetes, you can configure Consul service mesh and deploy mesh gateways for your Kubernetes cluster using the Consul Helm Chart. Learn more in the Consul Kubernetes documentation, as well as the related Kubernetes Mesh Gateways tutorial.

If you are using HCP Consul Dedicated, you will have to set up service mesh federation for the primary cluster. Use the guided process in the HCP Cloud portal. Afterwards, navigate to the Configure sidecar proxies section of this tutorial.

In this tutorial, you will configure Consul service mesh, replicate intentions across multiple datacenters, and use mesh gateways to enable inter-datacenter service communication.

Specifically, you will:

1. Enable Consul service mesh in both datacenters
2. Deploy two mesh gateways
3. Register services and sidecar proxies
4. Configure intentions, which are automatically replicated
5. Test that your services can communicate with each other

This tutorial configures cross-datacenter data plane communication to go through both the local and remote mesh gateway by using the local mesh gateway mode, as illustrated below. Other mesh gateway modes are supported.

Prerequisites

To complete this tutorial you will need two wide area network (WAN) joined Consul datacenters with access control list (ACL) replication enabled. If you are starting from scratch, follow these tutorials to set up your datacenters, or use them to check that you have the proper configuration:

• Deployment Guide
• Securing Consul with ACLs
• Enabling WAN Federation Control Plane Traffic

This tutorial labels the primary datacenter as primary and the secondary datacenter as secondary. Replace these labels with the names of your datacenters you would like to federate.

Enable ACL replication

You will also need to enable ACL replication, which you can do by following the ACL Replication for Multiple Datacenters tutorial with the following modification.

When creating the replication token for ACL management, it will need the following rules:

acl = "write"
operator = "write"
service_prefix "" {
  policy = "read"
  intentions = "read"
}

{
  "acl": "write",
  "operator": "write",
  "service_prefix": {
    "": {
      "policy": "read",
      "intentions": "read"
    }
  }
}

The replication token needs different permissions depending on what you want to accomplish. The above policy allows for ACL policy, role, and token replication with acl:write, CA replication with operator:write, and intention and configuration entry replication with service:*:read.

Install Envoy on Consul agents and mesh gateway machines

This tutorial uses Envoy as the sidecar proxies and mesh gateways. You will need to install it alongside your Consul clients. Each mesh gateway or sidecar proxy receives its configuration and updates from the local Consul client.

Verify Consul agent configuration

Each Consul agent should have a datacenter field in their agents. This identifies the server's datacenter.

datacenter = "<current datacenter>"

{
  "datacenter": "<current datacenter>"
}

Verify that each Consul agent has a bind_addr set to an IP address accessible across datacenters. The IP address can be public or private, as long as another datacenter can access it for RPC operations.

bind_addr = "<IP address accessible across datacenters>"

{
  "bind_addr": "<IP address accessible across datacenters>"
}

Verify firewall rules

You should also make sure the following ports are open on your firewall between each datacenter:

• 8300/tcp: all Consul agents and mesh gateways should allow traffic to each other for cross-DC WAN gossip across each datacenter. After initial federation, you can restrict this rule to traffic between a datacenter's agents and mesh gateway.

• 8302/tcp: all Consul agents and mesh gateways should allow traffic to each other for intra-DC WAN gossip across each datacenter. After initial federation, you can restrict this rule to traffic between a datacenter's agents and mesh gateway.

• 8302/udp: all Consul agents and mesh gateways should allow traffic to each other for intra-DC WAN gossip across each datacenter. After initial federation, you can restrict this rule to traffic between a datacenter's agents and mesh gateway.

• 8443/tcp: mesh gateways should allow traffic from Consul agents and other mesh gateways across each datacenter.

• Envoy Listener Port/tcp: sidecar proxies for each application must allow traffic from mesh gateways.

Enable Consul service mesh in both datacenters

Once you have your datacenters set up and ACL replication configured, enable Connect service mesh in each of them sequentially. Consul service mesh's certificate authority (CA) (which is distinct from the Consul certificate authority that you manage using the consul tls CLI) will automatically bootstrap as soon as a server with service mesh enabled becomes the server cluster's leader. You can also use Vault as a Connect CA.

Enable Consul service mesh in the primary datacenter

Enable Consul service mesh in the primary datacenter, the service mesh CA, and federation by mesh gateway by adding the following snippet to the server configuration for each of your servers.

connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}

primary_datacenter = "primary"

{
  "connect": {
    "enabled": true,
    "enable_mesh_gateway_wan_federation": true
  },
  "primary_datacenter": "primary"
}

Load the new configuration by restarting each server one at a time, making sure to maintain quorum. This will be a similar process to performing a rolling restart during upgrades.

Stop the first server by running the following leave command.

$ consul leave

Once the server shuts down, restart it, and make sure that it is healthy and rejoins the other servers. Repeat this process until you've restarted all the servers with Connect enabled.

Enable Consul service mesh in the secondary datacenter

Once Consul service mesh is enabled in the primary datacenter, follow the same process to enable it in the secondary datacenter. Enable WAN federation over mesh gateways and set the primary datacenter on each Consul server in the secondary datacenter.

primary_gateways = [ "<primary-mesh-gateway-private-ip>:<primary-mesh-gateway-port>", ... ]
connect {
  enabled = true
  enable_mesh_gateway_wan_federation = true
}

primary_datacenter = "primary"

{
  "primary_gateways": ["primary-mesh-gateway-private-ip:primary-mesh-gateway-port"],
  "connect": {
    "enabled": true,
    "enable_mesh_gateway_wan_federation": true
  },
  "primary_datacenter": "primary"
}

The primary_gateways setting is required in order to initiate replication from the primary datacenter. This setting is equivalent to retry_join_wan, which takes a list of addresses to attempt joining to WAN.

The primary_datacenter setting is required in order to enable ACL replication between datacenters. It specifies which datacenter will write intentions and act as the root CA for service mesh. Intentions are automatically replicated from the primary to secondary datacenters.

Deploy mesh gateways

Connect mesh gateways proxy requests from services in one datacenter to services in another, so you will need to deploy your mesh gateways on nodes that can reach each other over the network. As we mentioned in the prerequisites, you will need to make sure that both Envoy and Consul are installed on the mesh gateway nodes. You won't necessarily want to run any services on these nodes other than Consul and Envoy because they will have access to the WAN.

Generate tokens for the mesh gateways

You will need to generate a token for each mesh gateway that gives it read access to the entire catalog.

service_prefix "gateway" {
   policy = "write"
}
service_prefix "" {
   policy = "read"
}
node_prefix "" {
   policy = "read"
}
agent_prefix "" {
   policy = "read"
}

$ consul acl policy create \
  -name mesh-gateway \
  -rules @mesh-gateway-policy.hcl

{
  "service_prefix": {
    "gateway": {
      "policy": "write"
    },
    "": {
      "policy": "read"
    }
  },
  "node_prefix": {
    "": {
      "policy": "read"
    }
  },
  "agent_prefix": {
    "": {
      "policy": "read"
    }
  }
}

$ consul acl policy create \
  -name mesh-gateway \
  -rules @mesh-gateway-policy.json

Generate a token for the primary datacenter's gateway from the new policy.

$ consul acl token create \
  -description "mesh-gateway primary datacenter token" \
  -policy-name mesh-gateway

Now generate a token for the secondary datacenter's gateway.

$ consul acl token create \
  -description "mesh-gateway secondary datacenter token" \
  -policy-name mesh-gateway

You'll apply those tokens when you deploy the mesh gateways.

Mesh gateway service configuration

When configuring your mesh gateways, you will need to set the advertise address for the service to be reachable both locally and remotely.

The -address option defines the address and port to advertise for services within the local datacenter to use to reach the mesh gateway instance. The -wan-address option defines the address and port to advertise for services or mesh gateways within remote datacenters to use to reach the mesh gateway instance. This address must be externally accessible from the remote datacenter.

Deploy the mesh gateway for your primary datacenter

Register and start the mesh gateway in your primary datacenter with the command below.

$ consul connect envoy -gateway=mesh -register \
                     -expose-servers \
                     -service "gateway-primary" \
                     -address "<private address>:8443" \
                     -wan-address "<externally accessible address>:8443"\
                     -token=<token for the primary dc gateway>

This example starts the mesh gateway listener on port 8443. You can update this to your preferred port.

Deploy the mesh gateway for your secondary datacenter

Register and start the mesh gateway in your secondary datacenter with the following command.

$ consul connect envoy -gateway=mesh -register \
                     -expose-servers \
                     -service "gateway-secondary" \
                     -address "<private address>:8443" \
                     -wan-address "<externally accessible address>:8443"\
                     -token=<gateway token for the secondary dc gateway>

This example starts the mesh gateway listener on port 8443. You can update this to your preferred port.

Verify WAN federation via mesh gateways

After a few moments, your datacenters should federate. Check that WAN federation completed by examining the Consul members.

$ consul members -wan

Node                        Address            Status  Type    Build   Protocol  DC         Partition  Segment
ip-10-254-2-250.secondary   10.254.2.250:8302  alive   server  1.12.0  2         secondary  default    <all>
ip-10-255-2-250.primary     10.255.2.250:8302  alive   server  1.12.0  2         primary    default    <all>

Check that each datacenter can read the other's services. Log into to a Consul server in the primary datacenter and check it can reach the catalog in the secondary datacenter.

$ curl -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" localhost:8500/v1/catalog/services?dc=secondary

Log into to a Consul server in the secondary datacenter and check it can reach the catalog in the primary datacenter.

$ curl -H "X-Consul-Token:${CONSUL_HTTP_TOKEN}" localhost:8500/v1/catalog/services?dc=primary

Configure sidecar proxies to use the mesh gateways

Create a proxy-defaults centralized configuration file for all the sidecar proxies in both datacenters. This file will instruct the sidecar proxies to send all inter-datacenter traffic through the gateways.

Kind = "proxy-defaults"
Name = "global"
MeshGateway {
  mode = "local"
}

$ consul config write proxy-defaults.hcl

{
  "Kind": "proxy-defaults",
  "Name": "global",
  "MeshGateway": {
    "Mode": "local"
  }
}

$ consul config write proxy-defaults.json

Once this step is complete, you will have set up Consul service mesh with gateways across multiple datacenters. Now you are ready to register the services inside your service mesh.

Register services

You can configure a service to use a sidecar proxy by including a sidecar_proxy stanza in its registration file. For this tutorial, you can use socat to act as a back end application for a registered dummy service called web which will represent the client service. These are example names. If you have services that you would like to connect, feel free to use those names instead.

Configure Consul clients

If your service runs on a machine with a Consul client, verify that your Consul agent configuration includes a configuration for the primary_datacenter.

primary_datacenter = "primary"

{
  "primary_datacenter": "primary"
}

Register a back end service in one datacenter

In one datacenter register a back end service and add a sidecar proxy registration. To do this you will either create a new registration file or edit an existing one to include a sidecar proxy stanza.

If you have ACLs enabled, you first need to create a token for the service. The token will be granted the permissions defined by the ACL service identities template which allows the service and its proxy to register with Consul, and discover other services in the environment.

$ consul acl token create -service-identity=socat

Then provide the SecretID in the token field of the service registration.

If you are using socat as your back end service, you will create a for the socat service registration in the Consul agent's configuration directory containing the snippet below.

service {
  name = "socat"
  port = 8181
  token = "<socat token's secret ID>"
  connect {
    sidecar_service {}
  }
}

{
  "service": {
    "name": "socat",
    "port": 8181,
    "token": "<socat token's secret ID>",
    "connect": {
      "sidecar_service": {}
    }
  }
}

Note the token and connect.sidecar_service stanzas of the registration. These are the parameters you would add to an existing service registration if you are not using the above example socat definition.

Reload the client with the new or modified registration.

$ consul reload

Then start Envoy specifying which service it will proxy.

$ consul connect envoy -sidecar-for=socat -token=<socat token's secret ID>

If you are using socat as your example, start it now on the port you specified in your registration by running the following command.

$ socat -v tcp-l:8181,fork exec:"/bin/cat"

Check that the socat service is running by accessing it using netcat on the same node. It will echo back anything you type.

$ nc 127.0.0.1 8181

hello
hello
echo
echo

Stop the running netcat service by typing ctrl + c.

Register a front end service in the other datacenter

Now in your other datacenter, you will register a service (with a sidecar proxy) that calls your back end service. The registration will need to declare the back end service as an upstream. Like the back end service, you can use an example service, which will be called web, or append the connect stanza to an existing registration with some customization.

If you have ACLs enabled, you first need to create a token for the service. The token will be granted the permissions to allow the service and its proxy to register with Consul, and discover other services in the environment.

$ consul acl token create -service-identity=web

Then provide the SecretID in the token field of the service registration.

To use web as your front end service, create a service registration file in the Consul agent's configuration directory that contains the following snippet.

service {
  name = "web"
  port = 8080
  token = "<web token's secret ID>"
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "socat"
            datacenter = "primary"
            local_bind_port = 8181
          }
        ]
      }
    }
  }
}

{
  "service": {
    "name": "web",
    "port": 8080,
    "token": "<web token's secret ID>",
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [
            {
              "destination_name": "socat",
              "datacenter": "primary",
              "local_bind_port": 8181
            }
          ]
        }
      }
    }
  }
}

Note the Connect part of the registration, which specifies socat as an upstream. If you are using another service as a back end, replace socat with its name and port 8181 with the desired port.

Reload the client with the new or modified registration.

$ consul reload

Then start Envoy and specify which service it will proxy.

$ consul connect envoy -sidecar-for web -token=<web token's secret ID>

Configure intentions

Now that your services are configured to be part of Consul service mesh, you will need to create intentions to allow them to communicate. Add an intention to allow the front end service to access the back end service. For services named web and socat perform the following steps.

First, create a file for a config entry definition named intention-config.hcl.

Kind = "service-intentions"
Name = "socat"
Sources = [
  {
    Name   = "web"
    Action = "allow"
  }
]

From the same directory where you saved this file, run the following command on your terminal to initialize these intention rules.

$ consul config write intention-config.hcl

Config entry written: service-intentions/socat

Consul will automatically forward intentions created in the secondary datacenter to the primary datacenter, where the servers will write them, and then automatically replicate the written intentions back to the secondary datacenter.

Test the connection

Now that you have services deployed in your Consul service mesh, verify they can contact each other. If you have been using the example web and socat services, from the node and datacenter where you registered the web service, start netcat connecting to the web service's proxy on port 8181 and type something for it to echo.

$ nc 127.0.0.1 8181

hello
hello
echo
echo

Next steps

In this tutorial you configured two WAN-joined datacenters to use Consul service mesh, deployed gateways in each datacenter, and connected two services to each other across datacenters.

Gateways know where to route traffic because of Server Name Indication (SNI), where the client service sends the destination information as part of the TLS handshake. Because mesh gateways rely on TLS to discover the traffic's destination, they require Consul service mesh to route traffic.

Now that you've learned how to deploy mesh gateways to proxy inter-datacenter traffic, you can deploy multiple mesh gateways for redundancy or high availability. The mesh gateways and proxies will automatically round-robin load balance traffic between the mesh gateways in the remote datacenter.

Visit the Consul documentation for a full list of configurations for Consul service mesh, including mesh gateway configuration options and setting up WAN federation via mesh gateways.