Service mesh proxy configuration

This topic describes how to declare a service mesh proxy in a service definition. The kind must be declared and information about the service they represent must be provided to function as a Consul service mesh proxy.

Configuration

Configure a service mesh proxy using the following syntax:

name =  <name of the service>
kind = "connect-proxy"
proxy = {
  destination_service_name = "<name of the service that the proxy represents>"
  <additional proxy parameters> = "<additional parameter values>"
}
port = <port where services can discover and connect to proxied services>

{
  "name": "<name of the service>",
  "kind": "connect-proxy",
  "proxy": {
    "destination_service_name": "<name of the service that the proxy represents>",
    "<additional proxy parameters>" : "<additional parameter values>"
  },
  "port": <port where services can discover and connect to proxied services>
}

The following table describes the parameters that must be added to the service definition to declare the service as a proxy.

You can specify several additional parameters to configure the proxy to meet your requirements. See Proxy Parameters for additional information.

Example

In the following example, a proxy named redis-proxy is registered as a service mesh proxy. It proxies to the redis service and is available at port 8181. As a result, any service mesh clients searching for a mesh-capable endpoint for redis will find this proxy.

kind = "connect-proxy"
name = "redis-proxy"
port = 8181
proxy = {
  destination_service_name = "redis"
}

{
  "name": "redis-proxy",
  "kind": "connect-proxy",
  "proxy": {
    "destination_service_name": "redis"
  },
  "port": 8181
}

Sidecar proxy configuration

Many service mesh proxies are deployed as sidecars. Sidecar proxies are co-located with the single service instance they represent and proxy all inbound traffic to.

Specify the following parameters in the proxy code block to configure a sidecar proxy in its own service registration:

• destination_service_id: String value that specifies the ID of the service being proxied. Refer to the proxy parameters reference for details.
• local_service_port: Integer value that specifies the port that the proxy should use to connect to the local service instance. Refer to the proxy parameters reference for details.
• local_service_address: String value that specifies the IP address or hostname that the proxy should use to connect to the local service. Refer to the proxy parameters reference for details.

Refer to Deploy sidecar services for additional information about configuring service mesh proxies as sidecars.

Complete configuration example

The following example includes values for all available options when registering a proxy instance.

kind = "connect-proxy"
name = "redis-proxy"
port = 8181
proxy = {
  config = {}
  destination_service_id = "redis1"
  destination_service_name = "redis"
  expose = {}
  local_service_address = "127.0.0.1"
  local_service_port = 9090
  local_service_socket_path = "/tmp/redis.sock"
  mesh_gateway = {}
  mode = "transparent"
  transparent_proxy = {}
  upstreams = []
}

{
  "name": "redis-proxy",
  "kind": "connect-proxy",
  "proxy": {
    "destination_service_name": "redis",
    "destination_service_id": "redis1",
    "local_service_address": "127.0.0.1",
    "local_service_port": 9090,
    "local_service_socket_path": "/tmp/redis.sock",
    "mode": "transparent",
    "transparent_proxy": {},
    "config": {},
    "upstreams": [],
    "mesh_gateway": {},
    "expose": {}
  },
  "port": 8181
}

Proxy parameters

The following table describes all parameters that can be defined in the proxy block.

Upstream configuration reference

You can configure the service mesh proxy to create listeners for upstream services. The listeners enable the upstream service to accept requests. You can specify the following parameters to configure upstream service listeners.

Upstream configuration examples

Upstreams support multiple destination types. The following examples include information about each implementation. Note that the examples in this topic use snake case, which is a convention that separates words with underscores, because the format is supported in configuration files and API registrations.

destination_type = "service"
destination_name = "redis"
datacenter = "dc1"
local_bind_address = "127.0.0.1"
local_bind_port = 1234
local_bind_socket_path = "/tmp/redis_5678.sock"
local_bind_socket_mode = "0700"
mesh_gateway = {
    mode = "local"
}

{
  "destination_type": "service",
  "destination_name": "redis",
  "datacenter": "dc1",
  "local_bind_address": "127.0.0.1",
  "local_bind_port": 1234,
  "local_bind_socket_path": "/tmp/redis_5678.sock",
  "local_bind_socket_mode": "0700",
  "mesh_gateway": {
    "mode": "local"
  }
}

destination_type = "prepared_query"
destination_name = "database"
local_bind_address = "127.0.0.1"
local_bind_port = 1234
config = {}

{
  "destination_type": "prepared_query",
  "destination_name": "database",
  "local_bind_address": "127.0.0.1",
  "local_bind_port": 1234,
  "config": {}
}

destination_partition = "finance"
destination_namespace = "default"
destination_type = "service"
destination_name = "billing"
local_bind_port = 9090

{
  "destination_partition": "finance",
  "destination_namespace": "default",
  "destination_type": "service",
  "destination_name": "billing",
  "local_bind_port": 9090
}

destination_peer = "cloud-services"
destination_partition = "finance"
destination_namespace = "default"
destination_type = "service"
destination_name = "api"
local_bind_port = 9090

{
  "destination_peer": "cloud-services",
  "destination_partition": "finance",
  "destination_namespace": "default",
  "destination_type": "service",
  "destination_name": "api",
  "local_bind_port": 9090
}

Proxy modes

You can configure which mode a proxy operates in by specifying "direct" or "transparent" in the mode parameter. The proxy mode determines the how proxies direct traffic. This feature was added in Consul 1.10.0.

• transparent: In this mode, inbound and outbound application traffic is captured and redirected through the proxy. This mode does not enable the traffic redirection. It directs Consul to configure Envoy as if traffic is already being redirected.
• direct: In this mode, the proxy's listeners must be dialed directly by the local application and other proxies.

You can also specify an empty string (""), which configures the proxy to operate in the default mode. The default mode is inherited from parent parameters in the following order of precedence:

1. Proxy service's Proxy configuration
2. The service-defaults configuration for the service.
3. The global proxy-defaults.

The proxy will default to direct mode if a mode cannot be determined from the parent parameters.

Transparent proxy configuration reference

The following examples show additional configuration for transparent proxies.

Note that the examples in this topic use snake case, which is a convention that separates words with underscores, because the format is supported in configuration files and API registrations.

Configure a proxy listener for outbound traffic on port 22500

{
  "outbound_listener_port": 22500,
  "dialed_directly": true
}

• outbound_listener_port (int: 15001) - The port the proxy should listen on for outbound traffic. This must be the port where outbound application traffic is captured and redirected to.

• dialed_directly (bool: false) - Determines whether this proxy instance's IP address can be dialed directly by transparent proxies. Transparent proxies typically dial upstreams using the "virtual" tagged address, which load balances across instances. A database cluster with a leader is an example where dialing individual instances can be helpful. Cannot be used with upstreams which define a destination_peer.

  Note: Dynamic routing rules such as failovers and redirects do not apply to services dialed directly. Additionally, the connection is proxied using a TCP proxy with a connection timeout of 5 seconds.

Mesh gateway configuration reference

The following examples show all possible mesh gateway configurations.

Note that the examples in this topic use snake case, which is a convention that separates words with underscores, because the format is supported in configuration files and API registrations.

Using local and egress gateways in the local datacenter

{
  "mode": "local"
}

Direct to remote and ingress services in a remote datacenter

{
  "mode": "remote"
}

Disable a mesh gateway

{
  "mode": "none"
}

Specify the default mesh gateway mode

{
  "mode": ""
}

• mode (string: "") - This defines the mode of operation for how upstreams with a remote destination datacenter get resolved.
  • "local" - Mesh gateway services in the local datacenter will be used as the next-hop destination for the upstream connection.
  • "remote" - Mesh gateway services in the remote/target datacenter will be used as the next-hop destination for the upstream connection.
  • "none" - No mesh gateway services will be used and the next-hop destination for the connection will be directly to the final service(s).
  • "" - Default mode. The default mode will be "none" if no other configuration enables them. The order of precedence for setting the mode is
    1. Upstream
    2. Proxy Service's Proxy configuration
    3. The service-defaults configuration for the service.
    4. The global proxy-defaults.

Expose paths configuration reference

The following examples show possible configurations to expose HTTP paths through Envoy.

Exposing paths through Envoy enables a service to protect itself by only listening on localhost, while still allowing non-mesh-enabled applications to contact an HTTP endpoint. Some examples include: exposing a /metrics path for Prometheus or /healthz for kubelet liveness checks.

Note that the examples in this topic use snake case, which is a convention that separates words with underscores, because the format is supported in configuration files and API registrations.

Expose listeners in Envoy for health checks

The following example exposes Envoy listeners to HTTP and GRPC checks registered with the local Consul agent:

{
  "expose": {
    "checks": true
  }
}

Expose an HTTP listener

The following example exposes and HTTP listener in Envoy at port 21500 that routes to an HTTP server listening at port 8080:

{
  "expose": {
    "paths": [
      {
        "path": "/healthz",
        "local_path_port": 8080,
        "listener_port": 21500
      }
    ]
  }
}

Expose an HTTP2 listener

The following example expose an HTTP2 listener in Envoy at port 21501 that routes to a gRPC server listening at port 9090:

{
  "expose": {
    "paths": [
      {
        "path": "/grpc.health.v1.Health/Check",
        "protocol": "http2",
        "local_path_port": 9090,
        "listener_port": 21501
      }
    ]
  }
}

• checks (bool: false) - If enabled, all HTTP and gRPC checks registered with the agent are exposed through Envoy. Envoy will expose listeners for these checks and will only accept connections originating from localhost or Consul's advertise address. The port for these listeners are dynamically allocated from expose_min_port to expose_max_port. This flag is useful when a Consul client cannot reach registered services over localhost. One example is when running Consul on Kubernetes, and Consul agents run in their own pods.
• paths array<Path>: [] - A list of paths to expose through Envoy.
  • path (string: "") - The HTTP path to expose. The path must be prefixed by a slash. ie: /metrics.
  • local_path_port (int: 0) - The port where the local service is listening for connections to the path.
  • listener_port (int: 0) - The port where the proxy will listen for connections. This port must be available for the listener to be set up. If the port is not free then Envoy will not expose a listener for the path, but the proxy registration will not fail.
  • protocol (string: "http") - Sets the protocol of the listener. One of http or http2. For gRPC use http2.

Unix domain sockets

To connect to a service using a local Unix domain socket instead of a port, add local_bind_socket_path and optionally local_bind_socket_mode to the upstream config for a service. The following examples show additional configurations for Unix domain sockets:

upstreams = [
  {
    destination_name = "service-1"
    local_bind_socket_path = "/tmp/socket_service_1"
    local_bind_socket_mode = "0700"
  }
]

{
  "upstreams": [
    {
        "destination_name":  "service-1",
        "local_bind_socket_path": "/tmp/socket_service_1",
        "local_bind_socket_mode": "0700"
    }
  ]
}

Envoy creates a socket with the specified path and mode and connects to service-1.

The mode field is optional. When omitted, Envoy uses the default mode. This is not applicable for abstract sockets. Refer to the Envoy documentation for details.

Similarly to expose a service listening on a Unix Domain socket to the service mesh, use either the socket_path field in the service definition or the local_service_socket_path field in the proxy definition. These fields are analogous to the port and service_port fields in their respective locations.

services {
  name = "service-2"
  socket_path = "/tmp/socket_service_2"
}

{
  "services": {
    "name": "service-2",
    "socket_path": "/tmp/socket_service_2"
  }
}

Or in the proxy definition:

services {
  name = "socket_service_2"
  connect {
    sidecar_service {
      proxy {
        name = "service-2"
        local_service_socket_path = "/tmp/socket_service_2"
      }
    }
  }
}

{
  "services": [
    {
      "name": "socket_service_2",
      "connect": {
        "sidecar_service": {
          "proxy": {
            "name": "service-2",
            "local_service_socket_path": "/tmp/socket_service_2"
          }
        }
      }
    }
  ]
}

There is no mode field since the service is expected to create the socket it is listening on, not the Envoy proxy. Again, the socket_path and local_service_socket_path fields conflict with address/port and local_service_address/local_service_port configuration options.