Nomad Autoscaler Agent

The Nomad Autoscaler agent has a variety of parameters that can be specified via configuration files or command-line flags. Configuration files are written in HCL. The Nomad Autoscaler can read and combine parameters from multiple configuration files or directories to configure the agent.

Nomad Namespaces

The Nomad Autoscaler currently has limited support for Nomad Namespaces. The nomad configuration below supports specifying a namespace; if configured with a namespace, the Autoscaler will retrieve scaling policies and perform autoscaling only for jobs in that namespace. A future version will include support for multiple namespaces.

Nomad ACLs

The Nomad Autoscaler can be configured to interact with an ACL-enabled Nomad cluster. Nomad 0.11 includes the scale ACL policy disposition specifically for supporting the operations of the Nomad Autoscaler. Therefore, the following policy is sufficient for creating an ACL token that can be used by the autoscaler for fetching scaling policies and scaling jobs:

namespace "default" {
  policy = "scale"
}

Other APM and target plugins may require additional ACLs; see the plugin documentation for more information.

Load Order and Merging

The Nomad Autoscaler agent supports multiple configuration files, which can be provided using the -config CLI flag. The flag can accept either a file or folder. In the case of a folder, any .hcl and .json files in the folder will be loaded and merged in lexicographical order. Directories are not loaded recursively.

For example:

$ nomad-autoscaler agent -config=autoscaler.conf -config=/etc/nomad-autoscaler -config=extra.json

This will load configuration from autoscaler.conf, from .hcl and .json files under /etc/nomad-autoscaler, and finally from extra.json. As each file is processed, its contents are merged into the existing configuration. When merging, any non-empty values from the latest config file will append or replace parameters in the current configuration. An empty value means "" for strings, 0 for integer or float values, and false for booleans.

SIGHUP Reload

The Nomad Autoscaler agent supports handling the SIGHUP signal for reloading without the need for restarting the agent. When sending a SIGHUP signal to the agent process, the agent will perform the following actions.

• reload the contents of the scaling policy directory as defined by the -policy-dir parameter.

General Parameters

• log_level (string: "INFO") - Specify the verbosity level of Nomad Autoscaler's logs. Valid values include DEBUG, INFO, and WARN, in decreasing order of verbosity.

• log_json (bool: false) - Output logs in a JSON format.

• plugin_dir (string: "./plugins") - The plugin directory is used to discover Nomad Autoscaler plugins.

http Block

The http block configures the Nomad Autoscaler's HTTP endpoint.

http {
  bind_address = "10.0.0.10"
  bind_port    = 9999
}

http Parameters

• bind_address (string: "127.0.0.1") - The HTTP address that the server will bind to.

• bind_port (int: 8080) - The port that the server will bind to.

nomad Block

The nomad block configures the Nomad Autoscaler's Nomad client.

nomad {
  address = "http://my-nomad.systems:4646"
  region  = "esp-vlc-1"
}

nomad Parameters

• address (string: "http://127.0.0.1:4646") - The address of the Nomad server in the form of protocol://addr:port.

• region (string: "global") - The region of the Nomad servers to connect with.

• namespace (string: "") - The target namespace for queries and actions bound to a namespace.

• token (string: "") - The SecretID of an ACL token to use to authenticate API requests with.

• http_auth (string: "") - The authentication information to use when connecting to a Nomad API which is using HTTP authentication.

• ca_cert (string: "") - Path to a PEM encoded CA cert file to use to verify the Nomad server SSL certificate.

• ca_path (string: "") - Path to a directory of PEM encoded CA cert files to verify the Nomad server SSL certificate.

• client_cert (string: "") - Path to a PEM encoded client certificate for TLS authentication to the Nomad server.

• client_key (string: "") - Path to an unencrypted PEM encoded private key matching the client certificate.

• tls_server_name (string: "") - The server name to use as the SNI host when connecting via TLS.

• skip_verify (bool: false) - Do not verify TLS certificates. This is strongly discouraged.

policy Block

The policy block configures the Nomad Autoscaler's policy handling.

policy {
  dir              = "/opt/nomad-autoscaler/policies"
  default_cooldown = "2m"
}

policy Parameters

• dir (string: "") - The path to a directory used to load scaling policies.

• default_cooldown (string: "5m") - The default cooldown that will be applied to all scaling policies which do not specify a cooldown period.

• default_evaluation_interval (string: "10s") - The default evaluation interval that will be applied to all scaling policies which do not specify an evaluation interval.

policy_eval Block

The policy_eval block holds the configuration related to the policy evaluation process.

policy_eval {
  ack_timeout    = "10m"
  delivery_limit = 4

  workers = {
    cluster    = 2
    horizontal = 2
  }
}

policy_eval Parameters

• ack_timeout (string: "5m") - The time limit that an eval must be ACK'd before being considered NACK'd.

• delivery_limit (int: 1) - The maximum number of times a policy evaluation can be dequeued from the broker.

• workers (map<string|int>: [cluster:10,horizontal:10]) - The number of workers to initialize for each queue. Nomad Autoscaler supports cluster and horizontal map keys. Nomad Autoscaler Enterprise supports additional vertical_mem and vertical_cpu entries.

telemetry Block

The telemetry block configures the Nomad Autoscaler's publication of metrics and telemetry to third-party systems.

telemetry {
  disable_hostname = true
}

telemetry Parameters

Due to the number of provider-specific parameters to the telemetry block, parameters in this section are grouped by the telemetry provider.

Common

The following options are available on all telemetry configurations.

• disable_hostname (bool: false) - Specifies if gauge values should be prefixed with the local hostname.

• enable_hostname_label (bool: false) - Enable adding hostname to metric labels.

• collection_interval (duration: "1s") - Specifies the time interval at which the Nomad agent collects telemetry data.

statsite

These telemetry parameters apply to Statsite.

• statsite_address (string: "") - Specifies the address of a statsite server to forward metrics data to.

telemetry {
  statsite_address = "statsite.company.local:8125"
}

statsd

These telemetry parameters apply to StatsD.

• statsd_address (string: "") - Specifies the address of a statsd server to forward metrics to.

telemetry {
  statsd_address = "statsd.company.local:8125"
}

datadog

These telemetry parameters apply to DataDog statsd.

• dogstatsd_address (string: "") - Specifies the address of a DataDog statsd server to forward metrics to.

• dogstatsd_tags (list: []) - Specifies a list of global tags that will be added to all telemetry packets sent to DogStatsD. It is a list of strings, where each string looks like my_tag_name:my_tag_value.

telemetry {
  dogstatsd_address = "dogstatsd.company.local:8125"
  dogstatsd_tags    = ["my_tag_name:my_tag_value"]
}

prometheus

These telemetry parameters apply to Prometheus.

• prometheus_metrics (bool: false) - Specifies whether the agent should make Prometheus formatted metrics available at /v1/metrics?format=prometheus.

• prometheus_retention_time (string: "24h") - Specifies the amount of time that Prometheus metrics are retained in memory.

circonus

These telemetry parameters apply to Circonus.

• circonus_api_token (string: "") - Specifies a valid Circonus API Token used to create/manage check. If provided, metric management is enabled.

• circonus_api_app (string: "nomad-autoscaler") - Specifies a valid app name associated with the API token.

• circonus_api_url (string: "https://api.circonus.com/v2") - Specifies the base URL to use for contacting the Circonus API.

• circonus_submission_interval (string: "10s") - Specifies the interval at which metrics are submitted to Circonus.

• circonus_submission_url (string: "") - Specifies the check.config.submission_url field, of a Check API object, from a previously created HTTPTRAP check.

• circonus_check_id (string: "") - Specifies the Check ID (not check bundle) from a previously created HTTPTrap check. The numeric portion of the check._cid field in the Check API object.

• circonus_check_force_metric_activation (bool: false) - Specifies if force activation of metrics which already exist and are not currently active. If check management is enabled, the default behavior is to add new metrics as they are encountered. If the metric already exists in the check, it will not be activated. This setting overrides that behavior.

• circonus_check_instance_id (string: "<hostname>:<application>") - Serves to uniquely identify the metrics coming from this instance. It can be used to maintain metric continuity with transient or ephemeral instances as they move around within an infrastructure. By default, this is set to "hostname:application name" (e.g. host123:nomad-autoscaler).

• circonus_check_search_tag (string: <service>:<application>) - Specifies a special tag which, when coupled with the instance id, helps to narrow down the search results when neither a Submission URL or Check ID is provided. By default, this is set to "service:app" (e.g. service:nomad-autoscaler).

• circonus_check_display_name (string: "") - Specifies a name to give a check when it is created. This name is displayed in the Circonus UI Checks list.

• circonus_check_tags (string: "") - Comma separated list of additional tags to add to a check when it is created.

• circonus_broker_id (string: "") - Specifies the ID of a specific Circonus Broker to use when creating a new check. The numeric portion of broker._cid field in a Broker API object. If metric management is enabled and neither a Submission URL nor Check ID is provided, an attempt will be made to search for an existing check using Instance ID and Search Tag. If one is not found, a new HTTPTrap check will be created. By default, this is a random Enterprise Broker is selected, or, the default Circonus Public Broker.

• circonus_broker_select_tag (string: "") - Specifies a special tag which will be used to select a Circonus Broker when a Broker ID is not provided. The best use of this is to as a hint for which broker should be used based on where this particular instance is running (e.g., a specific geographic location or datacenter, dc:sfo).

apm Block

The apm block is used to configure application performance metric (APM) plugins.

apm "example-apm-plugin" {
  driver = "example-apm-plugin"
  args   = ["-my-flag"]

  config = {
    address = "http://127.0.0.1:9090"
  }
}

apm Parameters

• args (array<string>: []) - Specifies a set of arguments to pass to the plugin binary when it is executed.

• driver (string: "") - The plugin's executable name relative to to the plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.

• config (map<string><string>: nil) - Specifies configuration values for the plugin either as HCL or JSON. The accepted values are plugin specific. Please refer to the individual plugin's documentation.

target Block

The target block is used to configure scaling target plugins.

target "example-target-plugin" {
  driver = "example-target-plugin"
  args   = ["-my-flag"]

  config = {
    region = "esp-vlc-1"
  }
}

target Parameters

• args (array<string>: []) - Specifies a set of arguments to pass to the plugin binary when it is executed.

• driver (string: "") - The plugin's executable name relative to to the plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.

• config (map<string><string>: nil) - Specifies configuration values for the plugin either as HCL or JSON. The accepted values are plugin specific. Please refer to the individual plugin's documentation.

strategy Block

The strategy block is used to configure scaling strategy plugins.

strategy "example-strategy-plugin" {
  driver = "example-strategy-plugin"
  args   = ["-my-flag"]

  config = {
    algorithm = "complex"
  }
}

strategy Parameters

• args (array<string>: []) - Specifies a set of arguments to pass to the plugin binary when it is executed.

• driver (string: "") - The plugin's executable name relative to to the plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.

• config (map<string><string>: nil) - Specifies configuration values for the plugin either as HCL or JSON. The accepted values are plugin specific. Please refer to the individual plugin's documentation.