Client HTTP API

The /client endpoints are used to interact with the Nomad clients.

Both clients and servers can handle client endpoints. This is particularly useful for when a direct connection to a client is not possible due to the network configuration. For high volume access to the client endpoints, particularly endpoints streaming file contents, direct access to the node should be preferred as it avoids adding additional load to the servers.

When accessing the endpoints via the server, if the desired node is ambiguous based on the URL, an additional ?node_id query parameter must be provided to disambiguate.

The /node endpoints provide node information that are retried from Nomad servers.

Read Dynamic Node Metadata

This endpoint queries Node metadata on a specific Client agent and responds with the following fields:

Meta (object) - The Node metadata that will be registered with the Nomad servers and used by the scheduler (after up to 10 seconds of delay for batching). This is the merged version of the Static and Dynamic fields.
Static (object) - The Node metadata set in the Client agent's configuration file. Only loaded when an agent starts.
Dynamic (object) - The Node metadata set via the API (see below). Unlike Meta and Static, this object may contain null values to differentiate "unset" keys from keys with an empty string value ("").

Note that /v1/node/:node_id only contains the Meta object. It may take up to 10 seconds for dynamic Node metadata to be sent to Servers and visible through the Node API. Use the Node API to see the version of Node metadata the scheduler uses.

Method	Path	Produces
`GET`	`/v1/client/metadata`	`application/json`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`node:read`

Parameters

:node_id (string: <optional>) - Specifies the node to query. This is required when the endpoint is being accessed via a server. Defaults to the node recieving the request otherwise. This is specified as part of the URL. Note, this must be the full node ID, not the short 8-character one. This must be specified as part of the path (?node_id=...).

Sample Request

$ nomad operator api /v1/client/metadata

Sample Response

Formatted by appending ?pretty above.

{
    "Meta": {
        "connect.proxy_concurrency": "1",
        "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.log_level": "debug",
        "foo": "bar"
    },
    "Dynamic": {
        "key_to_unset": null,
        "foo": "bar",
        "connect.log_level": "debug"
    },
    "Static": {
        "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.log_level": "info",
        "connect.proxy_concurrency": "1"
    }
}

Sample Request

Update Dynamic Node Metadata

This endpoint updates dynamic Node metadata on a specific Client agent. Since dynamic Node metadata is only periodically synchronized to Nomad Servers, the Meta returned in this API may not be reflected in the /v1/node/:node_id API for up to 10 seconds. Scheduling uses the Node API version of Meta.

For convenience this endpoint returns the same response as a GET.

Method	Path	Produces
`POST`	`/v1/client/metadata`	`application/json`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`node:write`

Parameters

NodeID or :node_id (string: <optional>) - Specifies the node to query. This is required when the endpoint is being accessed via a server. Defaults to the node recieving the request otherwise. This is specified as part of the URL. Note, this must be the full node ID, not the short 8-character one. This may be specified as part of the path (?node_id=...) or request (NodeID: "...").
Meta (object: <required>) - Specifies the Node metadata keys to update. Only specified keys are updated.
- <key> (string: <optional>) - Specifies a metadata key to update to a particular value. Since "" is a valid value and distinct from unset, the null value is used to mark a key as unset. Keys must be valid dotted HCL identifiers. For example connect.log_level is a valid key while some/path is not.

Sample Payload

{
  "Meta": {
    "connect.log_level": "debug",
    "key_to_unset": null,
    "foo": "bar"
  }
}

Sample Request

Assuming the above payload is in a file called meta.json.

$ nomad operator api /v1/client/metadata < meta.json

Sample Response

Formatted by appending ?pretty above.

{
    "Meta": {
        "connect.proxy_concurrency": "1",
        "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.log_level": "debug",
        "foo": "bar"
    },
    "Dynamic": {
        "key_to_unset": null,
        "foo": "bar",
        "connect.log_level": "debug"
    },
    "Static": {
        "connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
        "connect.log_level": "info",
        "connect.proxy_concurrency": "1"
    }
}

Read Stats

This endpoint queries the actual resources consumed on a node. The API endpoint is hosted by the Nomad client and requests have to be made to the nomad client whose resource usage metrics are of interest.

Method	Path	Produces
`GET`	`/v1/client/stats`	`application/json`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`node:read`

Parameters

node_id (string: <optional>) - Specifies the node to query. This is required when the endpoint is being accessed via a server. This is specified as part of the URL. Note, this must be the full node ID, not the short 8-character one. This is specified as part of the path.

Sample Request

$ nomad operator api /v1/client/stats

Sample Response

{
  "AllocDirStats": {
    "Available": 142943150080,
    "Device": "",
    "InodesUsedPercent": 0.05312946180421879,
    "Mountpoint": "",
    "Size": 249783500800,
    "Used": 106578206720,
    "UsedPercent": 42.668233241448746
  },
  "CPU": [
    {
      "CPU": "cpu0",
      "Idle": 80,
      "System": 11,
      "Total": 20,
      "User": 9
    },
    {
      "CPU": "cpu1",
      "Idle": 99,
      "System": 0,
      "Total": 1,
      "User": 1
    },
    {
      "CPU": "cpu2",
      "Idle": 89,
      "System": 7.000000000000001,
      "Total": 11,
      "User": 4
    },
    {
      "CPU": "cpu3",
      "Idle": 100,
      "System": 0,
      "Total": 0,
      "User": 0
    },
    {
      "CPU": "cpu4",
      "Idle": 92.92929292929293,
      "System": 4.040404040404041,
      "Total": 7.07070707070707,
      "User": 3.0303030303030303
    },
    {
      "CPU": "cpu5",
      "Idle": 99,
      "System": 1,
      "Total": 1,
      "User": 0
    },
    {
      "CPU": "cpu6",
      "Idle": 92.07920792079209,
      "System": 4.9504950495049505,
      "Total": 7.920792079207921,
      "User": 2.9702970297029703
    },
    {
      "CPU": "cpu7",
      "Idle": 99,
      "System": 0,
      "Total": 1,
      "User": 1
    }
  ],
  "CPUTicksConsumed": 1126.8044804480448,
  "DeviceStats": [
    {
      "InstanceStats": {
        "6a61929e-d572-092d-5921-156a913f8e56": {
          "Stats": {
            "Attributes": {
              "Used Memory": {
                "Desc": "Memory in use by the device",
                "IntNumeratorVal": 128,
                "Unit": "MiB"
              }
            },
            "Nested": {}
          },
          "Summary": {
            "Desc": "Memory in use by the device",
            "IntNumeratorVal": 128,
            "Unit": "MiB"
          },
          "Timestamp": "2020-12-18T17:15:08.949806Z"
        }
      },
      "Name": "modelA",
      "Type": "skeleton",
      "Vendor": "hashicorp"
    },
    {
      "InstanceStats": {
        "73af5d3e-00f9-0786-9bc1-8f5ffa953f15": {
          "Stats": {
            "Attributes": {
              "Used Memory": {
                "Desc": "Memory in use by the device",
                "IntNumeratorVal": 128,
                "Unit": "MiB"
              }
            },
            "Nested": {}
          },
          "Summary": {
            "Desc": "Memory in use by the device",
            "IntNumeratorVal": 128,
            "Unit": "MiB"
          },
          "Timestamp": "2020-12-18T17:15:08.949806Z"
        }
      },
      "Name": "modelB",
      "Type": "skeleton",
      "Vendor": "hashicorp"
    }
  ],
  "DiskStats": [
    {
      "Available": 142943150080,
      "Device": "/dev/disk1",
      "InodesUsedPercent": 0.05312946180421879,
      "Mountpoint": "/",
      "Size": 249783500800,
      "Used": 106578206720,
      "UsedPercent": 42.668233241448746
    }
  ],
  "Memory": {
    "Available": 6232244224,
    "Free": 470618112,
    "Total": 17179869184,
    "Used": 10947624960
  },
  "Timestamp": 1495743032992498200,
  "Uptime": 193520
}

Read Allocation Statistics

The client allocation endpoint is used to query the actual resources consumed by an allocation.

Method	Path	Produces
`GET`	`/v1/client/allocation/:alloc_id/stats`	`application/json`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-job`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.

Sample Request

$ nomad operator api \
    /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/stats

Sample Response

{
  "ResourceUsage": {
    "CpuStats": {
      "Measured": ["Throttled Periods", "Throttled Time", "Percent"],
      "Percent": 0.14159538847117795,
      "SystemMode": 0,
      "ThrottledPeriods": 0,
      "ThrottledTime": 0,
      "TotalTicks": 3.256693934837093,
      "UserMode": 0
    },
    "MemoryStats": {
      "Cache": 1744896,
      "KernelMaxUsage": 0,
      "KernelUsage": 0,
      "MaxUsage": 4710400,
      "Measured": ["RSS", "Cache", "Swap", "Max Usage"],
      "RSS": 1486848,
      "Swap": 0
    }
  },
  "Tasks": {
    "redis": {
      "Pids": null,
      "ResourceUsage": {
        "CpuStats": {
          "Measured": ["Throttled Periods", "Throttled Time", "Percent"],
          "Percent": 0.14159538847117795,
          "SystemMode": 0,
          "ThrottledPeriods": 0,
          "ThrottledTime": 0,
          "TotalTicks": 3.256693934837093,
          "UserMode": 0
        },
        "MemoryStats": {
          "Cache": 1744896,
          "KernelMaxUsage": 0,
          "KernelUsage": 0,
          "MaxUsage": 4710400,
          "Measured": ["RSS", "Cache", "Swap", "Max Usage"],
          "RSS": 1486848,
          "Swap": 0
        }
      },
      "Timestamp": 1495743243970720000
    }
  },
  "Timestamp": 1495743243970720000
}

Read File

This endpoint reads the contents of a file in an allocation directory.

Method	Path	Produces
`GET`	`/v1/client/fs/cat/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
path (string: "/") - Specifies the path of the file to read, relative to the root of the allocation directory.

Sample Request

$ nomad operator api \
    /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99

$ nomad operator api \
    /v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=alloc/file.json

Sample Response

(whatever was in the file...)

Read File at Offset

This endpoint reads the contents of a file in an allocation directory at a particular offset and limit.

Method	Path	Produces
`GET`	`/v1/client/fs/readat/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
path (string: "/") - Specifies the path of the file to read, relative to the root of the allocation directory.
offset (int: <required>) - Specifies the byte offset from where content will be read.
limit (int: <required>) - Specifies the number of bytes to read from the offset.

Sample Request

$ nomad operator api \
    /v1/client/fs/readat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/foo&offset=1323&limit=19303

Sample Response

(whatever was in the file, starting from offset, up to limit bytes...)

Stream File

This endpoint streams the contents of a file in an allocation directory.

Method	Path	Produces
`GET`	`/v1/client/fs/stream/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
path (string: "/") - Specifies the path of the file to read, relative to the root of the allocation directory.
follow (bool: true)- Specifies whether to tail the file.
offset (int: <required>) - Specifies the byte offset from where content will be read.
origin (string: "start|end") - Applies the relative offset to either the start or end of the file.

Sample Request

$ nomad operator api \
    /v1/client/fs/stream/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/logs/redis.log

Sample Response

({
  "File": "alloc/logs/redis.log",
  "Offset": 3604480,
  "Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..."
},
{
  "File": "alloc/logs/redis.log",
  "FileEvent": "file deleted"
})

Field Reference

The return value is a stream of frames. These frames contain the following fields:

Data - A base64 encoding of the bytes being streamed.
FileEvent - An event that could cause a change in the streams position. The possible values are "file deleted" and "file truncated".
Offset - Offset is the offset into the stream.
File - The name of the file being streamed.

Stream Logs

This endpoint streams a task's stderr/stdout logs. Note that if logging is set to disabled=true for the task, this endpoint will return a 404 error.

Method	Path	Produces
`GET`	`/v1/client/fs/logs/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-logs` or `namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
task (string: <required>) - Specifies the name of the task inside the allocation to stream logs from.
follow (bool: false)- Specifies whether to tail the logs.
type (string: "stderr|stdout") - Specifies the stream to stream.
offset (int: 0) - Specifies the offset to start streaming from.
origin (string: "start|end") - Specifies either "start" or "end" and applies the offset relative to either the start or end of the logs respectively. Defaults to "start".
plain (bool: false) - Return just the plain text without framing. This can be useful when viewing logs in a browser.

Sample Request

$ nomad operator api \
    /v1/client/fs/logs/5fc98185-17ff-26bc-a802-0c74fa471c99

Sample Response

({
  "File": "alloc/logs/redis.stdout.0",
  "Offset": 3604480,
  "Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..."
},
{
  "File": "alloc/logs/redis.stdout.0",
  "FileEvent": "file deleted"
})

Field Reference

The return value is a stream of frames. These frames contain the following fields:

Data - A base64 encoding of the bytes being streamed.
FileEvent - An event that could cause a change in the streams position. The possible values are "file deleted" and "file truncated".
Offset - Offset is the offset into the stream.
File - The name of the file being streamed.

List Files

This endpoint lists files in an allocation directory.

Method	Path	Produces
`GET`	`/v1/client/fs/ls/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
path (string: "/") - Specifies the path of the file to read, relative to the root of the allocation directory.

Sample Request

$ nomad operator api \
    /v1/client/fs/ls/5fc98185-17ff-26bc-a802-0c74fa471c99

Sample Response

[
  {
    "Name": "alloc",
    "IsDir": true,
    "Size": 4096,
    "FileMode": "drwxrwxr-x",
    "ModTime": "2016-03-15T15:40:00.414236712-07:00"
  },
  {
    "Name": "redis",
    "IsDir": true,
    "Size": 4096,
    "FileMode": "drwxrwxr-x",
    "ModTime": "2016-03-15T15:40:56.810238153-07:00"
  }
]

Stat File

This endpoint stats a file in an allocation.

Method	Path	Produces
`GET`	`/v1/client/fs/stat/:alloc_id`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:read-fs`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.
path (string: "/") - Specifies the path of the file to read, relative to the root of the allocation directory.

Sample Request

$ nomad operator api \
    /v1/client/fs/stat/5fc98185-17ff-26bc-a802-0c74fa471c99

Sample Response

{
  "Name": "redis-syslog-collector.out",
  "IsDir": false,
  "Size": 96,
  "FileMode": "-rw-rw-r--",
  "ModTime": "2016-03-15T15:40:56.822238153-07:00"
}

GC Allocation

This endpoint forces a garbage collection of a particular, stopped allocation on a node. Note that the allocation will still exist on the server and appear in server responses.

Method	Path	Produces
`GET`	`/v1/client/allocation/:alloc_id/gc`	`application/json`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`namespace:submit-job`

Parameters

:alloc_id (string: <required>) - Specifies the allocation ID to query. This is specified as part of the URL. Note, this must be the full allocation ID, not the short 8-character one. This is specified as part of the path.

Sample Request

$ nomad operator api \
    /v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/gc

GC All Allocation

This endpoint forces a garbage collection of all stopped allocations on a node.

Method	Path	Produces
`GET`	`/v1/client/gc`	`text/plain`

The table below shows this endpoint's support for blocking queries and required ACLs.

Blocking Queries	ACL Required
`NO`	`node:write`

Parameters

node_id (string: <optional>) - Specifies the node to target. This is required when the endpoint is being accessed via a server. This is specified as part of the URL. Note, this must be the full node ID, not the short 8-character one. This is specified as part of the path.

Sample Request

$ nomad operator api /v1/client/gc