HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

consul Block

Placementconsul

The consul block configures the Nomad agent's communication with Consul for service discovery and key-value integration. When configured, tasks can register themselves with Consul, and the Nomad cluster can automatically bootstrap itself.

consul {
  address = "127.0.0.1:8500"
  auth    = "admin:password"
  token   = "abcd1234"
}

A default consul block is automatically merged with all Nomad agent configurations. These sensible defaults automatically enable Consul integration if Consul is detected on the system. This allows for seamless bootstrapping of the cluster with zero configuration. To put it another way: if you have a Consul agent running on the same host as the Nomad agent with the default configuration, Nomad will be able to automatically connect to the rest of the Nomad cluster.

If the local Consul agent is configured and accessible by the Nomad agents, the Nomad cluster will automatically bootstrap provided server_auto_join, client_auto_join, and auto_advertise are all enabled (which is the default).

An important requirement is that each Nomad agent talks to a unique Consul agent. Nomad agents should be configured to talk to Consul agents and not Consul servers. If you are observing flapping services, you may have multiple Nomad agents talking to the same Consul agent. As such avoid configuring Nomad to talk to Consul via DNS such as consul.service.consul

In Nomad Enterprise, you may specify multiple consul blocks to configure access to multiple Consul clusters. Each Consul cluster must have a different value for the name field.

consul Parameters

Some parameters are expected to be specified in the configuration file of Nomad agents running as clients, servers, or in all agents. Parameters are safely ignored if placed in a configuration file where they are not expected to be defined.

Parameters for Nomad Clients and Servers

These parameters should be defined in the configuration file of all Nomad agents.

  • address (string: "127.0.0.1:8500") - Specifies the address to the local Consul agent, given in the format host:port. Supports Unix sockets with the format: unix:///tmp/consul/consul.sock. Will default to the CONSUL_HTTP_ADDR environment variable if set. The value supports go-sockaddr/template format.

  • auth (string: "") - Specifies the HTTP Basic Authentication information to use for access to the Consul Agent, given in the format username:password.

  • auto_advertise (bool: true) - Specifies if Nomad should advertise its services in Consul. The services are named according to server_service_name and client_service_name. Nomad servers and clients advertise their respective services, each tagged appropriately with either http or rpc tag. Nomad servers also advertise a serf tagged service.

  • ca_file (string: "") - Specifies an optional path to the CA certificate used for Consul communication. This defaults to the system bundle if unspecified. Will default to the CONSUL_CACERT environment variable if set.

  • cert_file (string: "") - Specifies the path to the certificate used for Consul communication. If this is set then you need to also set key_file.

  • checks_use_advertise (bool: false) - Specifies if Consul health checks should bind to the advertise address. By default, this is the first HTTP address. If no HTTP address is specified, it will fall back to the bind_addr.

  • key_file (string: "") - Specifies the path to the private key used for Consul communication. If this is set then you need to also set cert_file.

  • name (string: "default") Enterprise - Specifies a name for the cluster so it can be referred to by job submitters in the job specification's consul.cluster or service.cluster fields. In Nomad Community Edition, only the "default" cluster will be used, so this field should be omitted.

  • namespace (string: "") Enterprise - Specifies the Consul namespace used by the Consul integration. If non-empty, this namespace will be used on all Consul API calls and for Consul Connect configurations, unless overridden by the job's consul.namespace field.

  • ssl (bool: false) - Specifies if the transport scheme should use HTTPS to communicate with the Consul agent. Will default to the CONSUL_HTTP_SSL environment variable if set.

  • tags (array<string>: []) - Specifies optional Consul tags to be registered with the Nomad server and client services.

  • timeout (string: "5s") - Specifies a time limit for requests made against Consul. This is specified using a label suffix like "10s".

  • token (string: "") - Specifies the token used to provide a per-request ACL token. This option overrides the Consul Agent's default token. If the token is not set here or on the Consul agent, it defaults to Consul's anonymous policy, which may or may not allow writes. Defaults to the CONSUL_HTTP_TOKEN environment variable if set. Nomad cannot refresh this token; if the token is deleted, Nomad is not able to communicate with Consul.

  • verify_ssl (bool: true)- Specifies if SSL peer verification should be used when communicating to the Consul API client over HTTPS. Will default to the CONSUL_HTTP_SSL_VERIFY environment variable if set.

Parameters for Nomad Clients

These parameters should only be defined in the configuration file of Nomad agents with client.enabled set to true.

  • client_auto_join (bool: true) - Specifies if the Nomad clients should automatically discover servers in the same region by searching for the Consul service name defined in the server_service_name option. The search occurs if the client is not registered with any servers or it is unable to heartbeat to the leader of the region, in which case it may be partitioned and searches for other servers.

  • client_service_name (string: "nomad-client") - Specifies the name of the service in Consul for the Nomad clients.

  • client_http_check_name (string: "Nomad Client HTTP Check") - Specifies the HTTP health check name in Consul for the Nomad clients.

  • client_failures_before_critical (int: 0) - Specifies the number of consecutive failures before the Nomad client Consul health check is critical.

  • client_failures_before_warning (int: 0) - Specifies the number of consecutive failures before the Nomad client Consul health check shows a warning.

  • grpc_address (string: "127.0.0.1:8502") - Specifies the address to the local Consul agent for gRPC requests, given in the format host:port. Note that Consul does not enable the grpc or grpc_tls listeners by default.

  • grpc_ca_file (string: "") - Specifies an optional path to the GRPC CA certificate used for communication between Connect sidecar proxies and Consul agents. Will default to the CONSUL_GRPC_CACERT environment variable if set.

    Warning

    Consul does not support incoming TLS verification of Envoy sidecars. You should set tls.grpc.verify_incoming = false in your Consul configuration when using Connect. See Consul/#13088 for more details.

  • share_ssl (bool: true) - Specifies whether the Nomad client should share its Consul SSL configuration with Connect Native applications. Includes values of ca_file, cert_file, key_file, ssl, and verify_ssl. Does not include the values for the ACL token or auth. This option should be disabled in environments where Consul ACLs are not enabled.

  • service_auth_method (string: "nomad-workloads") - Specifies the name of the Consul authentication method that will be used to login with a Nomad JWT for services.

  • task_auth_method (string: "nomad-workloads") - Specifies the name of the Consul authentication method that will be used to login with a Nomad JWT for tasks.

Parameters for Nomad Servers

These parameters should only be defined in the configuration file of Nomad agents with server.enabled set to true.

  • server_service_name (string: "nomad") - Specifies the name of the service in Consul for the Nomad servers.

  • server_http_check_name (string: "Nomad Server HTTP Check") - Specifies the HTTP health check name in Consul for the Nomad servers.

  • server_serf_check_name (string: "Nomad Server Serf Check") - Specifies the Serf health check name in Consul for the Nomad servers.

  • server_rpc_check_name (string: "Nomad Server RPC Check") - Specifies the RPC health check name in Consul for the Nomad servers.

  • server_auto_join (bool: true) - Specifies if the Nomad servers should automatically discover and join other Nomad servers by searching for the Consul service name defined in the server_service_name option. This search only happens if the server does not have a leader.

  • server_failures_before_critical (int: 0) - Specifies the number of consecutive failures before the Nomad server Consul health check is critical.

  • server_failures_before_warning (int: 0) - Specifies the number of consecutive failures before the Nomad server Consul health check shows a warning.

  • service_identity (Identity: nil) - Specifies a default Workload Identity to use when obtaining Service Identity tokens from Consul to register services. Refer to Workload Identity for a recommended configuration.

  • task_identity (Identity: nil) - Specifies a default Workload Identity to use when obtaining Consul tokens from Consul to support template blocks. Refer to Workload Identity for a recommended configuration.

Deprecated Parameters

These parameters are used by the deprecated token-based authentication flow and will be removed in a future release.

  • allow_unauthenticated (bool: true) - Specifies if users submitting jobs to the Nomad server should be required to provide their own Consul token, proving they have access to the service identity policies required by the Consul Connect enabled services listed in the job. This option should be disabled in an untrusted environment.

Workload Identity

The service_identity and task_identity blocks accept all the same values as the job specification's identity (except that the service_identity ignores the env and file fields). By ensuring these blocks are set on the Nomad servers, you can automatically migrate your existing jobs to use Workload Identity without modifying the job specification and resubmitting them.

The recommended configuration for service_identity and task_identity is as follows. See Migrating to Using Workload Identity with Consul for details on how to configure Consul to accept these identities. Note that the ttl field here refers to the TTL of the Nomad identity and not the Consul token.

consul {
  service_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }

  task_identity {
    aud = ["consul.io"]
    ttl = "1h"
  }
}

service_identity Parameters

  • aud (array<string>: []) - List of valid recipients for this workload identity. This value must match the BoundAudiences configuration in the Consul JWT auth method. It is recommended to provide one, and only one, audience to minimize where the identity may be used.

  • ttl (string: "") - Specifies for how long the workload identity should be considered as valid before expiring.

task_identity Parameters

  • aud (array<string>: []) - List of valid recipients for this workload identity. This value must match the BoundAudiences configuration in the Consul JWT auth method. It is recommended to provide one, and only one, audience to minimize where the identity may be used.

  • env (bool: false) - If true the workload identity will be available in the task's NOMAD_TOKEN_consul_default (or NOMAD_TOKEN_consul_<name> depending on the name field) environment variable.

  • file (bool: false) - If true the workload identity will be available in the task's filesystem via the path secrets/nomad_consul_default.jwt (or secrets/nomad_consul_<name>.jwt depending on the name field). If the task.user parameter is set, the token file will only be readable by that user. Otherwise the file is readable by everyone but is protected by parent directory permissions.

  • ttl (string: "") - Specifies for how long the workload identity should be considered as valid before expiring.

consul Examples

Default

This example shows the default Consul integration:

consul {
  address             = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
}

Custom Address and Port

This example shows pointing the Nomad agent at a different Consul address. Note that you should never point directly at a Consul server; always point to a local client. In this example, the Consul server is bound and listening on the node's private IP address instead of localhost, so we use that:

consul {
  address = "10.0.2.4:8500"
}

Custom SSL

This example shows configuring custom SSL certificates to communicate with the Consul agent. The Consul agent should be configured to accept certificates similarly, but that is not discussed here:

consul {
  ssl       = true
  ca_file   = "/var/ssl/bundle/ca.bundle"
  cert_file = "/etc/ssl/consul.crt"
  key_file  = "/etc/ssl/consul.key"
}

Consul ACL policy for Nomad

Nomad agents need access to Consul in order to register themselves in the service catalog and discover other Nomad agents via service discovery for automatic clustering. Nomad clients use Consul tokens from Workload Identity to register services and checks but need permissions on their own token to deregister. Nomad servers also create configuration entries for Consul Service Mesh, so the specific permissions vary slightly between Nomad servers and clients. The following Consul ACL policies represent the minimal permissions Nomad servers and clients need.

agent_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
}

acl  = "write"
mesh = "write"

Consul Namespace Enterprise

Consul does not allow ACL policies associated with namespaces to use agent permissions. Nomad requires agent:read permissions. In order to use the consul_namespace feature, Nomad will need a token generated in Consul's default namespace. That token should be created with agent:read as well as a namespace block with the other relevant permissions for running Nomad in the intended namespace. The Consul policy below shows an example policy configuration for a Nomad server:

agent_prefix "" {
  policy = "read"
}

namespace "nomad-ns" {
  acl = "write"

  key_prefix "" {
    policy = "read"
  }

  node_prefix "" {
    policy = "read"
  }

  service_prefix "" {
    policy = "write"
  }
}
Edit this page on GitHub