Connect nodes into a cluster

In order to create a Nomad cluster out of individual nodes, you need to introduce them to one another. There are several ways to perform this:

• Manually
• Cloud Auto-Join
• Consul

This tutorial describes each method and provides configuration snippets, which you can use as starting points for your own configuration.

Manual clustering

Manually bootstrapping a Nomad cluster does not rely on additional tooling, but does require operator participation in the cluster formation process. When bootstrapping, Nomad servers and clients must be started and informed with the address of at least one Nomad server.

As you can tell, this creates a chicken-and-egg problem where one server must first be fully bootstrapped and configured before the remaining servers and clients can join the cluster. This requirement can add additional provisioning time as well as ordered dependencies during provisioning.

First, you need to bootstrap a single Nomad server and capture its IP address. Place this address in the configuration once you have that nodes IP address.

For Nomad servers, this configuration may look something like this:

server {
  enabled          = true
  bootstrap_expect = 3

  # This is the IP address of the first server provisioned
  server_join {
    retry_join = ["<known-address>:4648"]
  }
}

Alternatively, you can supply a server's address after the servers have all been started by running the server join command on the servers individually to cluster the servers. All servers can join one other server, and then rely on the gossip protocol to discover the rest.

$ nomad server join <known-address>

For Nomad clients, the configuration may look something like:

client {
  enabled = true
  server_join {
    retry_join = ["<known-address>:4647"]
  }
}

The client node's server list can be updated at run time using the node config command.

$ nomad node config -update-servers <IP>:4647

The port corresponds to the RPC port. If no port is specified with the IP address, the default RPC port of 4647 is assumed.

As servers are added or removed from the cluster, this information is pushed to the client. This means only one server must be specified because, after initial contact, the full set of servers in the client's region are shared with the client.

Join nodes using cloud auto-join

As of Nomad 0.8.4, retry_join accepts a unified interface using the go-discover library for doing automatic cluster joining using cloud metadata. To use retry-join with a supported cloud provider, specify the configuration on the command line or configuration file as a key=value key=value ... string. Values are taken literally and must not be URL encoded. If the values contain spaces, backslashes or double quotes they need to be double quoted and the usual escaping rules apply.

{
  "retry_join": ["provider=my-cloud config=val config2=\"some other val\" ..."]
}

Consult the cloud provider-specific configurations in the cloud-autojoin documentation. This can be combined with static IP or DNS addresses or even multiple configurations for different providers. In order to use discovery behind a proxy, you will need to set HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables per Golang net/http library.

Use Consul to automatically cluster nodes

To automatically bootstrap a Nomad cluster, Nomad can leverage another HashiCorp open source tool, Consul. Bootstrapping Nomad is easiest against an existing Consul cluster. The Nomad servers and clients will become informed of each other's existence when the Consul agent is installed and configured on each host. As an added benefit, integrating Consul with Nomad provides service and health check registration for applications which later run under Nomad.

Consul models infrastructures as datacenters and multiple Consul datacenters can be connected over the WAN so that clients can discover nodes in other datacenters. Since Nomad regions can encapsulate many datacenters, you should be running a Consul cluster in every Nomad region and connecting them over the WAN. Refer to the Consul tutorial for both bootstrapping a single datacenter and connecting multiple Consul clusters over the WAN.

If a Consul agent is installed on the host prior to Nomad starting, the Nomad agent will register with Consul and discover other nodes.

For servers, you must inform the cluster how many servers you expect to have. This is required to form the initial quorum, since Nomad is unaware of how many peers to expect. For example, to form a region with three Nomad servers, you would use the following Nomad configuration file:

# /etc/nomad.d/server.hcl

# data_dir tends to be environment specific.
data_dir = "/opt/nomad/data"

server {
  enabled          = true
  bootstrap_expect = 3
}

This configuration would be saved to disk and then run:

$ nomad agent -config=/etc/nomad.d/server.hcl

A similar configuration is available for Nomad clients:

# /etc/nomad.d/client.hcl

datacenter = "dc1"

# data_dir tends to be environment specific.
data_dir = "/opt/nomad/data"

client {
  enabled = true
}

The agent is started in a similar manner:

$ sudo nomad agent -config=/etc/nomad.d/client.hcl

Nomad clients should always run as root (or with sudo). The above configurations include no IP or DNS addresses between the clients and servers. This is because Nomad detected the existence of Consul and utilized service discovery to form the cluster.

Consul auto-join internals

As discussed in the previous section, Nomad merges multiple configuration files together, so the -config may be specified more than once:

$ nomad agent -config=base.hcl -config=server.hcl

In addition to merging configuration on the command line, Nomad also maintains its own internal configurations (called "default configs") which include reasonable base defaults. One of those default configurations includes a "consul" block, which specifies reasonable defaults for connecting to and integrating with Consul. In essence, this configuration file resembles the following:

# You do not need to add this to your configuration file. This is an example
# that is part of Nomad's internal default configuration for Consul integration.
consul {
  # The address to the Consul agent.
  address = "127.0.0.1:8500"

  # The service name to register the server and client with Consul.
  server_service_name = "nomad"
  client_service_name = "nomad-client"

  # Enables automatically registering the services.
  auto_advertise = true

  # Enabling the server and client to bootstrap using Consul.
  server_auto_join = true
  client_auto_join = true
}

Refer to the consul stanza documentation for the complete set of configuration options.