Nomad Configuration

Nomad agents have a variety of parameters that can be specified via configuration files or command-line flags. Configuration files are written in HCL. Nomad can read and combine parameters from multiple configuration files or directories to configure the Nomad agent.

Load Order and Merging

The Nomad agent supports multiple configuration files, which can be provided using the -config CLI flag. The flag can accept either a file or folder. In the case of a folder, any .hcl and .json files in the folder will be loaded and merged in lexicographical order. Directories are not loaded recursively.

For example:

$ nomad agent -config=server.conf -config=/etc/nomad -config=extra.json

This will load configuration from server.conf, from .hcl and .json files under /etc/nomad, and finally from extra.json.

As each file is processed, its contents are merged into the existing configuration. When merging, any non-empty values from the latest config file will append or replace parameters in the current configuration. An empty value means "" for strings, 0 for integer or float values, and false for booleans. Since empty values are ignored you cannot disable a parameter like server mode once you've enabled it.

Please note that plugin blocks for a given plugin always replace and override each other, rather than being merged.

Here is an example Nomad agent configuration that runs in both client and server mode.

data_dir  = "/var/lib/nomad"

bind_addr = "0.0.0.0" # the default

advertise {
  # Defaults to the first private IP address.
  http = "1.2.3.4"
  rpc  = "1.2.3.4"
  serf = "1.2.3.4:5648" # non-default ports may be specified
}

server {
  enabled          = true
  bootstrap_expect = 3
}

client {
  enabled       = true
}

plugin "raw_exec" {
  config {
    enabled = true
  }
}

consul {
  address = "1.2.3.4:8500"
}

General Parameters

• acl (ACL: nil) - Specifies configuration which is specific to ACLs.

• addresses (Addresses: see below) - Specifies the bind address for individual network services. Any values configured in this block take precedence over the default bind_addr. These values should be specified in IP format without a port (ex. "0.0.0.0"). To set the port, see the ports field. The values support go-sockaddr/template format.

  • http - The address the HTTP server is bound to. This is the most common bind address to change. The http field accepts multiple values, separated by spaces, to bind to multiple addresses.

  • rpc - The address to bind the internal RPC interfaces to. Should be exposed only to other cluster members if possible.

  • serf - The address used to bind the gossip layer to. Both a TCP and UDP listener will be exposed on this address. Should be exposed only to other cluster members if possible.

• advertise (Advertise: see below) - Specifies the advertise address for individual network services. This can be used to advertise a different address to the peers of a server or a client node to support more complex network configurations such as NAT. This configuration is optional, and defaults to the bind address of the specific network service if it is not provided. Any values configured in this block take precedence over the default bind_addr.

  If the bind address is 0.0.0.0 then the IP address of the default private network interface advertised. The advertise values may include an alternate port, but otherwise default to the port used by the bind address. The values support go-sockaddr/template format.

  • http - The address to advertise for the HTTP interface. This should be reachable by all the nodes from which end users are going to use the Nomad CLI tools.

  • rpc - The address used to advertise to Nomad clients for connecting to Nomad servers for RPC. This allows Nomad clients to connect to Nomad servers from behind a NAT gateway. This address much be reachable by all Nomad client nodes. When set, the Nomad servers will use the advertise.serf address for RPC connections amongst themselves. Setting this value on a Nomad client has no effect.

  • serf - The address advertised for the gossip layer. This address must be reachable from all server nodes. It is not required that clients can reach this address. Nomad servers will communicate to each other over RPC using the advertised Serf IP and advertised RPC Port.

• audit (Audit: nil) - Enterprise-only. Specifies audit logging configuration.

• bind_addr (string: "0.0.0.0") - Specifies which address the Nomad agent should bind to for network services, including the HTTP interface as well as the internal gossip protocol and RPC mechanism. This should be specified in IP format, and can be used to easily bind all network services to the same address. It is also possible to bind the individual services to different addresses using the addresses configuration option. Dev mode (-dev) defaults to localhost. The value supports go-sockaddr/template format.

• client (Client: nil) - Specifies configuration which is specific to the Nomad client.

• consul (Consul: nil) - Specifies configuration for connecting to Consul.

• datacenter (string: "dc1") - Specifies the data center of the local agent. A datacenter is an abstract grouping of clients within a region. Clients are not required to be in the same datacenter as the servers they are joined with, but do need to be in the same region.

• data_dir (string: required) - Specifies a local directory used to store agent state. Client nodes use this directory by default to store temporary allocation data as well as cluster information. Server nodes use this directory to store cluster state, including the replicated log and snapshot data. This must be specified as an absolute path. Nomad will create the directory on the host, if it does not exist when the agent process starts.

• disable_anonymous_signature (bool: false) - Specifies if Nomad should provide an anonymous signature for de-duplication with the update check.

• disable_update_check (bool: false) - Specifies if Nomad should not check for updates and security bulletins. This defaults to true in Nomad Enterprise.

• enable_debug (bool: false) - Specifies if the debugging HTTP endpoints should be enabled. These endpoints can be used with profiling tools to dump diagnostic information about Nomad's internals.

• enable_syslog (bool: false) - Specifies if the agent should log to syslog. This option only works on Unix based systems. The log level inherits from the Nomad agent log set in log_level

• http_api_response_headers (map<string|string>: nil) - Specifies user-defined headers to add to the HTTP API responses.

• leave_on_interrupt (bool: false) - Specifies if the agent should leave when receiving the interrupt signal. By default, any stop signal to an agent (interrupt or terminate) will cause the agent to exit after ensuring its internal state is committed to disk as needed. If this value is set to true on a server agent, the server will notify other servers of their intention to leave the peer set. You should only set this value to true on server agents if the terminated server will never join the cluster again. If this value is set to true on a client agent and the client is configured with drain_on_shutdown, the client will drain its workloads before shutting down.

• leave_on_terminate (bool: false) - Specifies if the agent should leave when receiving the terminate signal. By default, any stop signal to an agent (interrupt or terminate) will cause the agent to exit after ensuring its internal state is committed to disk as needed. If this value is set to true on a server agent, the server will notify other servers of their intention to leave the peer set. You should only set this value to true on server agents if the terminated server will never join the cluster again. If this value is set to true on a client agent and the client is configured with drain_on_shutdown, the client will drain its workloads before shutting down.

• limits - This is a nested object that configures limits that are enforced by the agent. The following parameters are available:

  • https_handshake_timeout (string: "5s") - Configures the limit for how long the HTTPS server in both client and server agents will wait for a client to complete a TLS handshake. This should be kept conservative as it limits how many connections an unauthenticated attacker can open if tls.http = true is being used (strongly recommended in production). Default value is 5s. 0 disables HTTP handshake timeouts.

  • http_max_conns_per_client (int: 100) - Configures a limit of how many concurrent TCP connections a single client IP address is allowed to open to the agent's HTTP server. This affects the HTTP servers in both client and server agents. Default value is 100. 0 disables HTTP connection limits.

  • rpc_handshake_timeout (string: "5s") - Configures the limit for how long servers will wait after a client TCP connection is established before they complete the connection handshake. When TLS is used, the same timeout applies to the TLS handshake separately from the initial protocol negotiation. All Nomad clients should perform this immediately on establishing a new connection. This should be kept conservative as it limits how many connections an unauthenticated attacker can open if TLS is being using to authenticate clients (strongly recommended in production). When tls.rpc is true on servers, this limits how long the connection and associated goroutines will be held open before the client successfully authenticates. Default value is 5s. 0 disables RPC handshake timeouts.

  • rpc_max_conns_per_client (int: 100) - Configures a limit of how many concurrent TCP connections a single source IP address is allowed to open to a single server. Client agents do not accept RPC TCP connections directly and therefore are not affected. It affects both clients connections and other server connections. Nomad clients multiplex many RPC calls over a single TCP connection, except for streaming endpoints such as log streaming which require their own connection when routed through servers. A server needs at least 2 TCP connections (1 Raft, 1 RPC) per peer server locally and in any federated region. Servers also need a TCP connection per routed streaming endpoint concurrently in use. Only operators use streaming endpoints; as of 0.10.3 Nomad client code does not. A reasonably low limit significantly reduces the ability of an unauthenticated attacker to consume unbounded resources by holding open many connections. You may need to increase this if WAN federated servers connect via proxies or NAT gateways or similar causing many legitimate connections from a single source IP. Default value is 100 which is designed to support the majority of users. 0 disables RPC connection limits. 26 is the minimum as 20 connections are always reserved for non-streaming connections (Raft and RPC) to ensure streaming RPCs do not prevent normal server operation. This minimum may be lowered in the future when streaming RPCs no longer require their own TCP connection.

• log_level (string: "INFO") - Specifies the verbosity of logs the Nomad agent will output. Valid log levels include WARN, INFO, DEBUG, or TRACE in increasing order of verbosity.

• log_include_location (bool: false) - Include file and line information in each log line.

• log_json (bool: false) - Output logs in a JSON format.

• log_file (string: "") - Specifies the path for logging. If the path does not includes a filename, the filename defaults to nomad.log. This setting can be combined with log_rotate_bytes and log_rotate_duration for a fine-grained log rotation control.

• log_rotate_bytes (int: 0) - Specifies the number of bytes that should be written to a log before it needs to be rotated. Unless specified, there is no limit to the number of bytes that can be written to a log file.

• log_rotate_duration (duration: "24h") - Specifies the maximum duration a log should be written to before it needs to be rotated. Must be a duration value such as 30s.

• log_rotate_max_files (int: 0) - Specifies the maximum number of older log file archives to keep, not including the log file currently being written. If set to 0 no files are ever deleted. Note that the total number of log files, for each of stderr and stdout, will be 1 greater than the log_rotate_max_files value.

• name (string: [hostname]) - Specifies the name of the local node. This value is used to identify individual agents. When specified on a server, the name must be unique within the region.

• plugin_dir (string: "") - Specifies the directory to use for looking up plugins. When this parameter is empty, Nomad will generate the path using the top-level data_dir suffixed with plugins, like "/opt/nomad/plugins". This must be an absolute path.

• plugin (Plugin: nil) - Specifies configuration for a specific plugin. The plugin block may be repeated, once for each plugin being configured. The key of the block is the plugin's executable name relative to the plugin_dir.

• ports (Port: see below) - Specifies the network ports used for different services required by the Nomad agent.

  • http - The port used to run the HTTP server.

  • rpc - The port used for internal RPC communication between agents and servers, and for inter-server traffic for the consensus algorithm (raft).

  • serf - The port used for the gossip protocol for cluster membership. Both TCP and UDP should be routable between the server nodes on this port.

    The default values are:

    ports {
      http = 4646
      rpc  = 4647
      serf = 4648
    }
    

• region (string: "global") - Specifies the region the Nomad agent is a member of. A region typically maps to a geographic region, for example us, with potentially multiple zones, which map to datacenters such as us-west and us-east.

  Be Careful! Changing the region of an agent with data stored in its data_dir may result in overwrite and data loss in the new region so avoid modifying this value after the agent starts. If you must change it, make sure no local state remains from the previous region before running the agent again.

• sentinel (Sentinel: nil) - Specifies configuration for Sentinel policies.

• server (Server: nil) - Specifies configuration which is specific to the Nomad server.

• syslog_facility (string: "LOCAL0") - Specifies the syslog facility to write to. This has no effect unless enable_syslog is true.

• tls (TLS: nil) - Specifies configuration for TLS.

• vault (Vault: nil) - Specifies configuration for connecting to Vault.

Configuration Reload

You can send the Nomad process a SIGHUP signal to reload a limited subset of its configuration. The fields that currently support reloading are:

• log_level: the log level is reloaded but not any other logging configuration value.
• tls: note this only reloads the TLS configuration between Nomad agents (servers and clients), and not the TLS configuration for communication with Consul or Vault.
• vault: note this only reloads the TLS configuration between Nomad and Vault, but not other configuration values.

In order to reload any other configuration values, you must restart the Nomad agent.

If the Nomad agent receives a SIGHUP during initialization, it may crash (see GH-3885). Ensure that the Nomad agent is able to receive RPC traffic before attempting to reload its configuration.

Examples

Custom Region and Datacenter

This example shows configuring a custom region and data center for the Nomad agent:

region     = "europe"
datacenter = "ams"

Enable CORS

This example shows how to enable CORS on the HTTP API endpoints:

http_api_response_headers {
  "Access-Control-Allow-Origin" = "*"
}