Consul service mesh in production

Consul service mesh can secure all inter-service communication with mutual TLS. It's designed to work with minimal configuration out of the box on platforms including Kubernetes or VMs. However, completing the security checklist and understanding the Consul security model are prerequisites for production deployments.

After completing this tutorial, you will be able to configure Consul service mesh to secure services. First, you will secure your Consul datacenter with ACLs and TLS encryption. Next, you will enable Consul service mesh on the servers and host. Finally, you will configure your services to be part of Consul service mesh.

The steps you need to get to a secure Consul datacenter with service mesh enabled are:

• Configure ACLs
• Configure agent transport encryption
• Bootstrap Certificate Authority for Consul service mesh
• Setup host firewall
• Configure service instances
• Next steps

For existing Consul deployments, it may be necessary to incrementally adopt Consul service mesh service-by-service. In this case, step one and two should already be complete. However, we recommend reviewing all steps since the final deployment goal is to be compliant with all the security recommendations in this tutorial.

Configure ACLs

Consul service mesh security is based on service identity. In practice, the identity of the service is only enforceable with sufficiently restrictive ACLs.

This section will not replace reading the full Secure Consul with ACLs tutorial but will highlight the specific requirements Consul service mesh relies on to ensure its security properties.

A service's identity, in the form of an x.509 certificate, will only be issued to an API client that has service:write permission for that service. In other words, any client that has permission to register an instance of a service will be able to identify as that service and access all of the resources that that service is allowed to access.

A secure ACL setup must meet the following criteria.

1. ACL default policy must be deny. If for any reason you cannot use the default policy of deny, you must add an explicit ACL denying anonymous service:write. Note, in this case the intention graph will also default to allow and explicit deny intentions will be needed to restrict service access. Also note that explicit rules to limit who can manage intentions are necessary in this case. It is assumed for the remainder of this tutorial that ACL policy defaults to deny.
2. Each service must have a unique ACL token that is restricted to service:write only for the named service. You can review the Securing Consul with ACLs tutorial for a service token example. Note, it is best practices for each instance to get a unique token as described below.

In practice, managing per-instance tokens requires automated ACL provisioning, for example using HashiCorp's Vault.

Configure agent transport encryption

Consul's gossip (UDP) and RPC (TCP) communications need to be encrypted otherwise attackers may be able to see ACL tokens while in flight between the server and client agents (RPC) or between client agent and application (HTTP). Certificate private keys never leave the host they are used on but are delivered to the application or proxy over local HTTP so local agent traffic should be encrypted where potentially untrusted parties might be able to observe localhost agent API traffic.

Follow the Gossip Encryption tutorial to ensure both gossip encryption is enabled in your Consul datacenter and the Secure Agent Communication with TLS Encryption tutorial to ensure RPC, HTTP and consensus communications are configured securely.

Bootstrap Certificate Authority for Consul service mesh

Consul service mesh comes with a built-in Certificate Authority (CA) that will bootstrap by default when you first enable Consul service mesh on your servers.

To use the built-in CA, enable it in the server's configuration.

connect {
  enabled = true
}

This configuration change requires a Consul server restart, which you can perform one server at a time to maintain availability in an existing datacenter.

When a server is enabled with Consul Service mesh and becomes the leader, it will bootstrap a new CA and generate it's own private key which is written to the Raft state.

Alternatively, an external private key can be provided via the CA configuration.

Setup host firewall

In order to enable inbound connections to Consul agents and the sidecar proxies, you may need to configure host or network firewalls to allow incoming connections to necessary ports.

In addition to Consul agent's communication ports any proxies will need to have ports open to accept incoming connections.

If using sidecar service registration Consul will by default assign ports from a configurable range. The default range is 21000 - 21255.

It is possible to prevent automated port selection by configuring sidecar_min_port and sidecar_max_port to both be 0, forcing any sidecar service registrations to need an explicit port configured.

It then becomes the same problem as opening ports necessary for any other application and might be managed by configuration management or a scheduler.

Configure service instances

With necessary ACL tokens in place, all service registrations need to have an appropriate ACL token present.

For on-disk configuration the token parameter of the service definition must be set.

service {
    name = "cassandra_db"
    port = 9002
    token = "<your_token_here>"
}

{
  "service": {
    "name": "cassandra_db",
    "port": 9002,
    "token": "<your_token_here>"
  }
}

For registration via the API the token is passed in the request header, X-Consul-Token, or by using the Go client configuration.

To avoid the overhead of a proxy, applications may natively integrate with Consul service mesh.

For examples of proxy service definitions see the proxy documentation.

Next steps

After securing your Consul datacenter with ACLs and TLS encryption, you can use Consul service mesh to secure service-to-service communication. If you encounter any issues while setting up Consul, there are many community resources where you can find help.