Storing Nomad Variables

Nomad Variables provide the option to securely store configuration at file-like paths directly in Nomad's state store. The contents of these secrets are encrypted and replicated between servers via raft. Access to secrets is controlled by ACL policies, and tasks have implicit ACL policies that allow them to access their own secrets. You can create, read, update, or delete secrets via the command line or in the Nomad web UI.

Note that the Nomad Variables feature is intended for small pieces of configuration data needed by workloads. Because writing to the Nomad state store uses resources needed by Nomad, it's not well-suited for large or fast-changing data. For example, do not store batch job results as variables - these should be stored in an external database. Variables are also not intended to be a full replacement for HashiCorp Vault. If you need powerful options like dynamic secrets or transit encryption, continue using Vault.

For complete documentation on the Nomad Variables feature and related concepts, see the Variables reference documentation, the Key Management documentation, and the Workload Identity documentation

In this tutorial you'll store variables in Nomad via the command line.

Using the variables command-line interface

First, create two namespaces named prod and dev.

$ nomad namespace apply -description "production environment" prod
Successfully applied namespace "prod"!

$ nomad namespace apply -description "development environment" dev
Successfully applied namespace "dev"!

Create a variable in the prod namespace.

nomad var put -namespace prod project/example user=me password=passw0rd1

Read the variable.

$ nomad var get -namespace prod project/example
Namespace   = prod
Path        = project/example
Create Time = 2022-09-19T11:21:23-04:00
Check Index = 27

Items
password = passw0rd1
user     = me

When writing a variable, you can use the -out option to display the output.

$ nomad var put -namespace prod -out table project/another-example user=you password=passw0rd2
Namespace   = prod
Path        = project/another-example
Create Time = 2022-09-19T11:21:56-04:00
Check Index = 28

Items
password = passw0rd2
user     = you

The var put and var get commands both accept the -out option. You can read variables in a machine-friendly format using the -out json option:

$ nomad var get -namespace prod -out json project/example
{
  "Namespace": "prod",
  "Path": "project/example",
  "CreateIndex": 27,
  "ModifyIndex": 27,
  "CreateTime": 1663600883130070567,
  "ModifyTime": 1663600883130070567,
  "Items": {
    "password": "passw0rd1",
    "user": "me"
  }
}

Updating variables

You can use var put to overwrite all items in a variable, but to avoid conflicting with other writes that may have happened since you last read the variable, you must use the -check-index flag and set it to the last modified index.

$ nomad var put -namespace prod -out json -check-index 27 project/example password=passw0rd3
{
  "Namespace": "prod",
  "Path": "project/example",
  "CreateIndex": 27,
  "ModifyIndex": 30,
  "CreateTime": 1663600883130070567,
  "ModifyTime": 1663600984406100205,
  "Items": {
    "password": "passw0rd3"
  }
}

Try it again, this time leaving off the -check-index flag (or setting it to an index value in the future). Note the value is not updated and you receive the current value to avoid conflicts.

$ nomad var put -namespace prod -out json -check-index 100 project/example password=passw0rd3

Check-and-Set conflict

    Your provided check-index (100) does not match the server-side index (30). The
    server-side item was last updated on 2022-09-19T11:23:04-04:00. If you are sure
    you want to perform this operation, add the -force or -check-index=30 flag
    before the positional arguments.

To patch a variable without overwriting all the values, you can pipe var get into var put with the initial value set to - to signal that you're accepting input from stdin. This will automatically set the -check-index flag so that you avoid conflicts.

$ nomad var get -namespace prod project/example | \
    nomad var put -namespace prod -out json - user=me2

The command will output the updated variable in JSON format.

  "Namespace": "prod",
  "Path": "project/example",
  "CreateIndex": 27,
  "ModifyIndex": 39,
  "CreateTime": 1663600883130070567,
  "ModifyTime": 1663601343585644939,
  "Items": {
    "password": "passw0rd3",
    "user": "me2"
  }
}

List the two variables. Note that the list response only includes the paths and metadata, not any of the data.

$ nomad var list -namespace prod
Namespace  Path                     Last Updated
prod       project/another-example  2022-09-19T11:21:56-04:00
prod       project/example          2022-09-19T11:29:03-04:00

List the variables, using a prefix filter. Note that only variables at the path matching the prefix are shown.

$ nomad var list -namespace prod project/ex
Namespace  Path             Last Updated
prod       project/example  2022-09-19T11:29:03-04:00

Create three more variables, this time in the dev namespace.

nomad var put -namespace dev system/config foo=bar
nomad var put -namespace dev project/example foo=bar
nomad var put -namespace dev another-project/example foo=bar

Using the wildcard namespace indicator (*), list all the variables you have access to. For many shells, the * character is significant, so you might need to wrap it in double (") or single (') quotation marks.

$ nomad var list -namespace '*'
Namespace  Path                     Last Updated
dev        another-project/example  2022-09-19T11:29:54-04:00
dev        project/example          2022-09-19T11:29:54-04:00
dev        system/config            2022-09-19T11:29:54-04:00
prod       project/another-example  2022-09-19T11:21:56-04:00
prod       project/example          2022-09-19T11:29:03-04:00

You can list the variables in a given namespace by using the namespace option. For example, list all of the variables in the dev namespace by adding -namespace dev to the nomad var list command.

$ nomad var list -namespace dev
Namespace  Path                     Last Updated
dev        another-project/example  2022-09-19T11:29:54-04:00
dev        project/example          2022-09-19T11:29:54-04:00
dev        system/config            2022-09-19T11:29:54-04:00

Next steps

Now that you have used the nomad var command to create and view Nomad Variables, consider learning more about how to control access to Nomad Variables with the Nomad Variables Access Control tutorial and how to use Nomad Variables in your jobs with the Accessing Variables From Tasks tutorial.