HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Nomad Home

server_join Block

Placementserver -> server_join
client -> server_join

The server_join block specifies how the Nomad agent will discover and connect to Nomad servers.

server_join {
  retry_join     = [ "1.1.1.1", "2.2.2.2" ]
  retry_max      = 3
  retry_interval = "15s"
}

server_join Parameters

  • retry_join (array<string>: []) - Specifies a list of Nomad server addresses and Cloud Auto-join configurations that are joined as cluster members. This is similar to start_join, but join attempts are retried up to retry_max times. Further, retry_join is available to both Nomad servers and clients, while start_join is only defined for Nomad servers. This is useful for cases where we know the address will become available eventually. Use retry_join with an array as a replacement for start_join, do not use both options.

    Server addresses must conform to the server address format.

    server_join {
  retry_join = [ "1.1.1.1", "2.2.2.2" ]
}

    Auto-join configurations must conform to the Cloud Auto-join format.

    Cloud-Auto join using go-discover to join an AWS EC2 cluster.

    server_join {
  retry_join = [ "provider=aws tag_key=..." ]
}

    Cloud-Auto join using go-netaddrs to join a Hetzner cluster of servers labeled with nomad-server=true using the hcloud CLI.

    server_join {
  retry_join = [ "exec=hcloud server list -o noheader -o columns=ipv4 -l nomad-server=true | tr '\n' ' '" ]
}

    Server addresses and Cloud Auto-join configurations may be used together. This is a mixed configuration containing a server address and Cloud Auto-join configuration.

    server_join {
  retry_join = [ "1.1.1.1", "provider=aws tag_key=..." ]
}

    See server address format for more information about expected server address formats and Cloud Auto-join for more information on expected Cloud Auto-join formats.

  • retry_interval (string: "30s") - Specifies the time to wait between retry join attempts.

  • retry_max (int: 0) - Specifies the maximum number of join attempts to be made before exiting with a return code of 1. By default, this is set to 0 which is interpreted as infinite retries.

  • start_join (array<string>: []) - Specifies a list of server addresses to join on startup. If Nomad is unable to join with any of the specified addresses, agent startup will fail. See the server address format section for more information on the format of the string. This field is defined only for Nomad servers and will result in a configuration parse error if included in a client configuration.

Server Address Format

This section describes the acceptable syntax and format for describing the location of a Nomad server. There are many ways to reference a Nomad server, including directly by IP address and resolving through DNS.

Directly via IP Address

It is possible to address another Nomad server using its IP address. This is done in the ip:port format, such as:

1.2.3.4:5678

If the IP address is an IPv6 address, it must be in URL format surrounded by brackets. For example:

[2001:db8::1]:5678

If the port option is omitted, it defaults to the Serf port, which is 4648 unless configured otherwise:

1.2.3.4 => 1.2.3.4:4648

Via Domains or DNS

It is possible to address another Nomad server using its DNS address. This is done in the address:port format, such as:

nomad-01.company.local:5678

If the port option is omitted, it defaults to the Serf port, which is 4648 unless configured otherwise:

nomad-01.company.local => nomad-01.company.local:4648

Via the go-discover interface

retry_join accepts a unified interface using the go-discover library for doing automated cluster joining using cloud provider metadata. See Cloud Auto-join for more information.

"provider=aws tag_key=..." => 1.2.3.4:4648

Cloud Auto-join

retry_join's Cloud Auto-join allows Nomad to automatically discover cluster server addresses using cloud provider metadata. Cloud Auto-join allows both go-discover and go-netaddrs formats.

Configurations prefixed with provider= use go-discover whereas configurations prefixed with exec= use go-netaddrs. For cloud providers not supported by go-discover, use go-netaddrs.

go-discover Configurations

go-discover configurations are prefixed with provider=.

The following sections describe the Cloud Auto-join retry_join options that are specific to a subset of supported cloud providers. For information on all providers, see further documentation in go-discover.

Amazon EC2

This returns the first private IP address of all servers in the given region which have the given tag_key and tag_value.

{
  "retry_join": ["provider=aws tag_key=... tag_value=..."]
}
  • provider (required) - the name of the provider ("aws" in this case).
  • tag_key (required) - the key of the tag to auto-join on.
  • tag_value (required) - the value of the tag to auto-join on.
  • region (optional) - the AWS region to authenticate in.
  • addr_type (optional) - the type of address to discover: private_v4, public_v4, public_v6. Default is private_v4. (>= 1.0)
  • access_key_id (optional) - the AWS access key for authentication (see below for more information about authenticating).
  • secret_access_key (optional) - the AWS secret access key for authentication (see below for more information about authenticating).
Authentication & Precedence

  • Static credentials access_key_id=... secret_access_key=...

  • Environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)

  • Shared credentials file (~/.aws/credentials or the path specified by AWS_SHARED_CREDENTIALS_FILE)

  • ECS task role metadata (container-specific).

  • EC2 instance role metadata.

    The only required IAM permission is ec2:DescribeInstances, and it is recommended that you make a dedicated key used only for auto-joining. If the region is omitted it will be discovered through the local instance's EC2 metadata endpoint.

Microsoft Azure

This returns the first private IP address of all servers in the given region which have the given tag_key and tag_value in the tenant and subscription, or in the given resource_group of a vm_scale_set for Virtual Machine Scale Sets.

{
  "retry_join": [
    "provider=azure tag_name=... tag_value=... tenant_id=... client_id=... subscription_id=... secret_access_key=..."
  ]
}
  • provider (required) - the name of the provider ("azure" in this case).
  • tenant_id (required) - the tenant to join machines in.
  • client_id (required) - the client to authenticate with.
  • secret_access_key (required) - the secret client key.

Use these configuration parameters when using tags:

  • tag_name - the name of the tag to auto-join on.
  • tag_value - the value of the tag to auto-join on.

Use these configuration parameters when using Virtual Machine Scale Sets (Consul 1.0.3 and later):

  • resource_group - the name of the resource group to filter on.

  • vm_scale_set - the name of the virtual machine scale set to filter on.

    When using tags the only permission needed is the ListAll method for NetworkInterfaces. When using Virtual Machine Scale Sets the only role action needed is Microsoft.Compute/virtualMachineScaleSets/*/read.

Google Compute Engine

This returns the first private IP address of all servers in the given project which have the given tag_value.

{
  "retry_join": ["provider=gce project_name=... tag_value=..."]
}
  • provider (required) - the name of the provider ("gce" in this case).
  • tag_value (required) - the value of the tag to auto-join on.
  • project_name (optional) - the name of the project to auto-join on. Discovered if not set.
  • zone_pattern (optional) - the list of zones can be restricted through an RE2 compatible regular expression. If omitted, servers in all zones are returned.
  • credentials_file (optional) - the credentials file for authentication. See below for more information.
Authentication & Precedence
  • Use credentials from credentials_file, if provided.
  • Use JSON file from GOOGLE_APPLICATION_CREDENTIALS environment variable.
  • Use JSON file in a location known to the gcloud command-line tool.
  • On Windows, this is %APPDATA%/gcloud/application_default_credentials.json.
  • On other systems, $HOME/.config/gcloud/application_default_credentials.json.
  • On Google Compute Engine, use credentials from the metadata server. In this final case any provided scopes are ignored.

Discovery requires a GCE Service Account. Credentials are searched using the following paths, in order of precedence.

go-netaddrs Configurations

go-netaddrs configurations are prefixed with exec=.

What follows the exec= prefix may be any executable program and its arguments. Commands run by go-netaddrs must return a list of space-delimited IPv4 or IPv6 addresses and exit with code 0 on success and non-zero on failure.

You can refer to the executable by its absolute file system path, or by the executable's name if it can be found on the Nomad agent's PATH. See executables in the current directory for more details on the lookup behavior.

Example go-netaddrs configuration.

server_join {
  retry_join = [ "exec=hcloud server list -o noheader -o columns=ipv4 -l nomad-server=true | tr '\n' ' '" ]
}

Here, hcloud's newline-delimited output has been re-formatted as space- delimited by piping its output into tr.

Output of an example go-netaddrs executable

"1.1.1.1 2.2.2.2"

Visit go-netaddrs for more information on go-netaddrs configuration.

Edit this page on GitHub