HashiConf 2024 Now streaming live from Boston! Attend for free
Nomad
Vault

Vault ACL with Nomad Workload Identities

  • 22min
  • |
  • Nomad
  • Vault
  • Interactive

Nomad integrates with Vault to retrieve static and dynamic secrets for workloads.

Production deployments of Nomad and Vault must always run with the Access Control List (ACL) system enabled since it protects against unauthorized access to the cluster. When ACLs are enabled, both Nomad and Vault must be properly configured in order for their integrations to work.

Nomad can generate workload identities for tasks, which are represented as JSON Web Tokens (JWT) signed by Nomad. These identities can be used as proof to third parties that a workload was actually created and is managed by Nomad. If the third party is configured to trust Nomad, it can automatically grant specific access and permissions to Nomad workloads.

In this tutorial, you will:

  • Start a Nomad and Vault agent with ACL enabled.
  • Generate ACL tokens to access Nomad and Vault.
  • Configure Vault to accept workload identities from Nomad.
  • Configure Nomad to automatically generate and sign workload identities for tasks that need access to Vault.
  • Deploy sample Nomad jobs that interact with Vault.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

This tutorial requires you to have basic familiarity with Nomad and Vault. If you are new to these tools, complete the Nomad Get Started and Vault Get Started tutorials before following this one.

You will need the following tools installed:

Start the Vault agent

Start a Vault dev server.

$ vault server -dev
==> Vault server configuration:

Administrative Namespace:
             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
   Environment Variables: CLICOLOR, COLORTERM, COMMAND_MODE, EDITOR, GODEBUG, GOPATH, HOME, LANG, LC_ALL, LOGNAME, LSCOLORS, LaunchInstanceID, OLDPWD, PATH, PWD, SECURITYSESSIONID, SHELL, SHLVL, SSH_AUTH_SOCK, TERM, TERM_PROGRAM, TERM_PROGRAM_VERSION, TMPDIR, USER, WINDOWID, XPC_FLAGS, XPC_SERVICE_NAME, _, __CFBundleIdentifier, __CF_USER_TEXT_ENCODING
              Go Version: go1.21.3
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level:
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.15.2, built 2023-11-06T11:33:28Z
             Version Sha: cf1b5cafa047bc8e4a3f93444fcb4011593b92cb

==> Vault server started! Log data will stream in below:

2023-11-20T20:08:27.583-0500 [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-11-20T20:08:27.583-0500 [INFO]  incrementing seal generation: generation=1
...
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variables:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: ZiFnmXBoD4fCy6S+SEIZ65fFHBz+yh4UBoI+9GJmRxk=
Root Token: hvs.HyI6cqHMcJK1dSk0LPuNEbKP

Development mode should NOT be used in production installations!

Dev Agents

This tutorial uses development agents for Vault and Nomad as a quick way to get started. Dev agents have ephemeral state and should not be used in production environments. They also run in the foreground of your terminal so do not close the terminal window or you will need to rerun the agent configuration steps again.

Copy the value for Root Token.

Open another terminal window in the same directory and set the root token as the environment variable VAULT_TOKEN. This terminal will act as the main terminal session where you will run commands.

$ export VAULT_TOKEN=...

Set the environment variable VAULT_ADDR.

$ export VAULT_ADDR='http://127.0.0.1:8200'

Start the Nomad agent

Create a file named nomad.hcl. Add the following contents to it and save the file.

nomad.hcl

acl {
  enabled = true
}

vault {
  enabled = true
  address = "http://127.0.0.1:8200"

  default_identity {
    aud = ["vault.io"]
    ttl = "1h"
  }
}

The vault block in this configuration file provides the information necessary for Nomad to connect to Vault.

It also defines a default workload identity, default_identity, that is automatically added to jobs that need access to Vault. Without this identity, you would need to define an identity block in your jobs for every task that needs access to Vault.

Open another terminal window in the same directory and start the Nomad dev agent.

$ sudo nomad agent -dev -config 'nomad.hcl'
==> Loaded configuration from nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
                Client: true
             Log Level: DEBUG
               Node Id: af9bef00-2e83-b704-2d6c-c62bc005431f
                Region: global (DC: dc1)
                Server: true
               Version: 1.7.0

==> Nomad agent started! Log data will stream in below:
...

Return to your main terminal window and bootstrap the Nomad ACL system.

$ nomad acl bootstrap
Accessor ID  = d1de8625-8556-0932-a25c-3aa71bfc0134
Secret ID    = 7f10099a-936c-3f3a-8783-f0980493e54b
Name         = Bootstrap Token
Type         = management
Global       = true
Create Time  = 2023-11-16 01:09:26.565422 +0000 UTC
Expiry Time  = <none>
Create Index = 23
Modify Index = 23
Policies     = n/a
Roles        = n/a

Copy the value of Secret ID and set it as the environment variable NOMAD_TOKEN.

$ export NOMAD_TOKEN=...

Using Bootstrap Tokens

The initial ACL bootstrap tokens from Vault and Nomad have full access to the cluster and should not be used for regular day-to-day operations in a production environment. They are used in this tutorial as an illustration. We recommend creating ACL policies and tokens with only a level of access necessary to perform the required operations.

Configure Vault to accept Nomad workload identities

Create a Vault ACL auth method

Enable a jwt auth method in Vault under the path jwt-nomad.

$ vault auth enable -path 'jwt-nomad' 'jwt'
Success! Enabled jwt auth method at: jwt-nomad/

The JWT auth method creates an endpoint that Nomad clients use to exchange Nomad workload identity JWTs for Vault ACL tokens.

Create a file named vault-auth-method-jwt-nomad.json. Add the following contents to it and save the file.

vault-auth-method-jwt-nomad.json

{
  "jwks_url": "http://127.0.0.1:4646/.well-known/jwks.json",
  "jwt_supported_algs": ["RS256", "EdDSA"],
  "default_role": "nomad-workloads"
}

This configuration file contains important information.

  • jwks_url is the URL that Vault uses to contact Nomad and retrieve the data necessary to validate Nomad workload identities. In a production environment, this should resolve to multiple Nomad agents via a reverse proxy, load balancer, or DNS entry to prevent a single point of failure.
  • default_role is the Vault ACL role applied by default to tokens generated by this auth method. You will create the nomad-workloads role in the next section.

Apply the configuration file vault-auth-method-jwt-nomad.json to the jwt-nomad auth method.

$ vault write auth/jwt-nomad/config '@vault-auth-method-jwt-nomad.json'
Success! Data written to: auth/jwt-nomad/config

Create a Vault ACL role

Create a file named vault-role-nomad-workloads.json. Add the following contents to it and save the file.

vault-role-nomad-workloads.json

{
  "role_type": "jwt",
  "bound_audiences": ["vault.io"],
  "user_claim": "/nomad_job_id",
  "user_claim_json_pointer": true,
  "claim_mappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task"
  },
  "token_type": "service",
  "token_policies": ["nomad-workloads"],
  "token_period": "30m",
  "token_explicit_max_ttl": 0
}

It defines properties for the Vault ACL tokens that are used for Nomad tasks.

  • bound_audiences configures Vault to only accept JWTs that have an audience value of vault.io. It must match the aud value present in the Nomad agent configuration and jobs.
  • claim_mappings are values from the Nomad workload identity. You will reference them when creating the Vault ACL policy for this role.
  • token_period determines how long the token is valid for before it expires. Nomad automatically renews tokens before they expire.
  • token_explicit_max_ttl is the maximum amount of time the token is valid. It must be set to 0 so Nomad can renew them for as long as the workload runs.
  • token_policies are the ACL policies applied to the tokens. They specify the permissions that tokens with this role have. You will create the policy nomad-workloads policy in the next section.

Create a Vault ACL role named nomad-workloads using the vault-role-nomad-workloads.json file.

$ vault write auth/jwt-nomad/role/nomad-workloads '@vault-role-nomad-workloads.json'
Success! Data written to: auth/jwt-nomad/role/nomad-workloads

This role is applied by default to Vault ACL tokens generated from the auth method jwt-nomad.

Create a Vault ACL policy

List all of the auth methods registered in Vault.

$ vault auth list
Path          Type     Accessor               Description                Version
----          ----     --------               -----------                -------
jwt-nomad/    jwt      auth_jwt_d34481ad      n/a                        n/a
token/        token    auth_token_510d42ca    token based credentials    n/a

Copy the Accessor value for the auth method in the jwt-nomad/ path.

Create a file named vault-policy-nomad-workloads.hcl. Add the following contents to it, replace all the instances of the AUTH_METHOD_ACCESSOR placeholder with the Accessor value from the output, and save the file. There are five occurrences to update.

vault-policy-nomad-workloads.hcl

path "kv/data/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_namespace}}/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}/*" {
  capabilities = ["read"]
}

path "kv/data/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_namespace}}/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}" {
  capabilities = ["read"]
}

path "kv/metadata/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_namespace}}/*" {
  capabilities = ["list"]
}

path "kv/metadata/*" {
  capabilities = ["list"]
}

This file is a templated Vault ACL policy that automatically grants Nomad workloads access to secrets based on the properties mapped in the claim_mappings of the Vault ACL role.

More specifically, this policy grants access to secrets in the path kv/data/<job namespace>/<job name>/* where <job namespace> and <job name> are dynamically set for each workload.

Create a Vault ACL policy named nomad-workloads using the file vault-policy-nomad-workloads.hcl.

$ vault policy write 'nomad-workloads' 'vault-policy-nomad-workloads.hcl'
Success! Uploaded policy: nomad-workloads

Run a Nomad job to read secrets from Vault

Start a MongoDB database with a root password that is read from Vault using the Nomad workload identity for the task.

Enable the kv secret engine.

$ vault secrets enable -version '2' 'kv'
Success! Enabled the kv secrets engine at: kv/

Write a secret in Vault for the database root password in the path default/mongo/config.

$ vault kv put -mount 'kv' 'default/mongo/config' 'root_password=secret-password'
======== Secret Path ========
kv/data/default/mongo/config

======= Metadata =======
Key                Value
---                -----
created_time       2023-11-21T02:52:42.061092Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create a file named mongo.nomad.hcl. Add the following contents to it and save the file.

mongo.nomad.hcl

job "mongo" {
  namespace = "default"

  group "db" {
    network {
      port "db" {
        static = 27017
      }
    }

    service {
      provider = "nomad"
      name     = "mongo"
      port     = "db"
    }

    task "mongo" {
      driver = "docker"

      config {
        image = "mongo:7"
        ports = ["db"]
      }

      vault {}

      template {
        data        = <<EOF
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD={{with secret "kv/data/default/mongo/config"}}{{.Data.data.root_password}}{{end}}
EOF
        destination = "secrets/env"
        env         = true
      }
    }
  }
}
  • The vault block indicates that the task needs access to Vault and that Nomad should use the task workload identity to get a Vault ACL token.
  • The template block reads the root password secret from Vault under the path kv/data/default/mongo/config.
  • The job runs in the Nomad default namespace and the job name is mongo. The nomad-workloads Vault role grants access to secrets in the path kv/data/default/mongo/*, which is where the root password exists.
  • The job does not specify any identity for Vault, so Nomad uses the default_identity from the agent configuration.

Run the job with the mongo.nomad.hcl file and wait for the deployment to complete.

$ nomad job run 'mongo.nomad.hcl'
==> 2023-11-20T22:09:16-05:00: Monitoring evaluation "34e72e7a"
    2023-11-20T22:09:16-05:00: Evaluation triggered by job "mongo"
    2023-11-20T22:09:17-05:00: Evaluation within deployment: "919f658b"
    2023-11-20T22:09:17-05:00: Allocation "49fc68d8" created: node "b8db12d2", group "db"
    2023-11-20T22:09:17-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-11-20T22:09:17-05:00: Evaluation "34e72e7a" finished with status "complete"
==> 2023-11-20T22:09:17-05:00: Monitoring deployment "919f658b"
  ✓ Deployment "919f658b" successful

    2023-11-20T22:09:28-05:00
    ID          = 919f658b
    Job ID      = mongo
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    db          1        1       1        0          2023-11-20T22:19:26-05:00

Verify that you are able to execute a query against the database using the root user credentials.

$ nomad alloc exec "$(nomad job allocs -t '{{with (index . 0)}}{{.ID}}{{end}}' 'mongo')" mongosh --username 'root' --password 'secret-password' --eval 'db.runCommand({connectionStatus : 1})' --quiet
{
  authInfo: {
    authenticatedUsers: [ { user: 'root', db: 'admin' } ],
    authenticatedUserRoles: [ { role: 'root', db: 'admin' } ]
  },
  ok: 1
}

Retrieve the job definition from Nomad and filter the output to only display its task.

$ nomad job inspect -t '{{sprig_toPrettyJson (index (index .TaskGroups 0).Tasks 0)}}' 'mongo'
{
  "Name": "mongo",
  "Driver": "docker",
  "User": "",
  "Lifecycle": null,
  "Config": {
    "image": "mongo:7",
    "ports": [
      "db"
    ]
  },
...

The Identities list contains the workload identity that Nomad injects following the specification in the default_identity block from the Nomad server configuration file.

{
  "Name": "mongo",
  "Driver": "docker",
  "User": "",
  "Lifecycle": null,
  "Config": {
    "image": "mongo:7",
    "ports": [
      "db"
    ]
  },
...
  "Identities": [
    {
      "Name": "vault_default",
      "Audience": [
        "vault.io"
      ],
      "ChangeMode": "",
      "ChangeSignal": "",
      "Env": false,
      "File": false,
      "ServiceName": "",
      "TTL": 3600000000000
    }
  ],
  "Actions": null
}

By configuring Vault to accept workload identities from Nomad, the Nomad task was able to automatically receive a Vault ACL token scoped to the level of access defined by the auth method default role. With this method, Nomad agents no longer require long-lived, highly permissive Vault tokens.

Configure Vault dynamic secrets for MongoDB

The MongoDB database reads a secret from Vault using the permissions in the default Vault ACL role.

In some situations, it may be necessary to customize the permissions granted to a task to be different from this default. This is done by creating additional Vault ACL roles for Nomad jobs.

Configure Vault with a dynamic secret for a non-root MongoDB user that the default Vault ACL policy does not grant access to then run a Nomad job using a custom Vault ACL role to be able to access the dynamic secret.

Enable the Vault database secrets engine.

$ vault secrets enable database
Success! Enabled the database secrets engine at: database/

Create a file named vault-dynamic-secret-mongo.json. Add the following contents to it and save the file.

vault-dynamic-secret-mongo.json

{
    "plugin_name": "mongodb-database-plugin",
    "allowed_roles": "mongo",
    "connection_url": "mongodb://{{username}}:{{password}}@127.0.0.1:27017/admin",
    "username": "root",
    "password": "secret-password"
}

Write the vault-dynamic-secret-mongo.json configuration for the MongoDB dynamic secret to connect to the database.

$ vault write 'database/config/mongo' '@vault-dynamic-secret-mongo.json'
Success! Data written to: database/config/mongo

Create a file named vault-database-role-mongo.json. Add the following contents to it and save the file.

vault-database-role-mongo.json

{
    "db_name": "mongo",
    "creation_statements": "{ \"db\": \"admin\", \"roles\": [{ \"role\": \"readWrite\" }, {\"role\": \"read\", \"db\": \"foo\"}] }",
    "default_ttl": "1h",
    "max_ttl": "24h"
}

Create a Vault database role using the vault-database-role-mongo.json file.

$ vault write 'database/roles/mongo' '@vault-database-role-mongo.json'
Success! Data written to: database/roles/mongo

Create a Vault ACL role to access the dynamic secret

Create a file named vault-role-mongo-dynamic-secret.json. Add the following contents to it and save the file.

vault-role-mongo-dynamic-secret.json

{
  "role_type": "jwt",
  "bound_audiences": ["vault.io"],
  "bound_claims": {
     "nomad_namespace": "default",
     "nomad_job_id": "mongo-query"
  },
  "user_claim": "/nomad_job_id",
  "user_claim_json_pointer": true,
  "claim_mappings": {
    "nomad_namespace": "nomad_namespace",
    "nomad_job_id": "nomad_job_id",
    "nomad_task": "nomad_task"
  },
  "token_type": "service",
  "token_policies": ["mongo-dynamic-secret"],
  "token_period": "30m",
  "token_explicit_max_ttl": 0
}

This role is similar to the one from earlier in the tutorial but this one uses a different policy, called mongo-dynamic-secret, which you will create in the next section.

It also defines a set of bound_claims to restrict which workload identities from Nomad are able to use this role. In this example, the role only allows the job mongo-query in the Nomad namespace default to use it.

Create the mongo-dynamic-secret ACL role using the vault-role-mongo-dynamic-secret.json file.

$ vault write 'auth/jwt-nomad/role/mongo-dynamic-secret' '@vault-role-mongo-dynamic-secret.json'
Success! Data written to: auth/jwt-nomad/role/mongo-dynamic-secret

Create a file named vault-policy-mongo-dynamic-secret.hcl. Add the following contents to it and save the file.

vault-policy-mongo-dynamic-secret.hcl

path "database/creds/mongo" {
  capabilities = ["read"]
}

This ACL policy only grants access to the specific path database/creds/mongo, which is not included in the default role used by the jwt-nomad auth method.

Create the mongo-dynamic-secret ACL policy using the vault-policy-mongo-dynamic-secret.hcl file.

$ vault policy write 'mongo-dynamic-secret' 'vault-policy-mongo-dynamic-secret.hcl'
Success! Uploaded policy: mongo-dynamic-secret

Run a Nomad job with a custom Vault ACL role

Create a file named mongo-query.nomad.hcl. Add the following contents to it and save the file.

mongo-query.nomad.hcl

job "mongo-query" {
  namespace = "default"
  type      = "batch"

  group "mongo-query" {
    task "mongo-query" {
      driver = "docker"

      config {
        image   = "mongo:7"
        command = "mongosh"
        args = [
          "--username", "${MONGO_USERNAME}",
          "--password", "${MONGO_PASSWORD}",
          "--eval", "db.runCommand({connectionStatus : 1})",
          "--quiet",
          "${MONGO_URL}",
        ]
      }

      vault {
        role = "mongo-dynamic-secret"
      }

      template {
        data        = <<EOF
{{with secret "database/creds/mongo"}}
MONGO_USERNAME={{.Data.username}}
MONGO_PASSWORD={{.Data.password}}
{{end}}
{{range nomadService 1 (env "NOMAD_ALLOC_ID") "mongo"}}
MONGO_URL=mongodb://{{.Address}}:{{.Port}}
{{end}}
EOF
        destination = "secrets/env"
        env         = true
      }
    }
  }
}

Note that the vault block specifies the mongo-dynamic-secret role. The task can only access the dynamic credentials for MongoDB.

The template block reads these credentials and exposes them as environment variables to the task so it can use them.

Run the Nomad job from the file mongo-query.nomad.hcl.

$ nomad job run 'mongo-query.nomad.hcl'
==> 2023-11-20T23:11:35-05:00: Monitoring evaluation "5fa56c67"
    2023-11-20T23:11:35-05:00: Evaluation triggered by job "mongo-query"
    2023-11-20T23:11:36-05:00: Allocation "909f0184" created: node "b8db12d2", group "mongo-query"
    2023-11-20T23:11:36-05:00: Evaluation status changed: "pending" -> "complete"
==> 2023-11-20T23:11:36-05:00: Evaluation "5fa56c67" finished with status "complete"

Retrieve the allocation information and wait until the status is complete. You may need to run the command a few times before the status changes to complete.

$ nomad job allocs 'mongo-query'
ID        Node ID   Task Group   Version  Desired  Status    Created  Modified
909f0184  b8db12d2  mongo-query  0        run      complete  25s ago  24s ago

Retrieve the query result to confirm the job was able to connect to the MongoDB database.

$ nomad alloc logs $(nomad job allocs -t '{{with (index . 0)}}{{.ID}}{{end}}' mongo-query)
{
  authInfo: {
    authenticatedUsers: [
      {
        user: 'v-jwt-nomad-mongo-mongo-QbF7HWHjwOi6PJWmNo1x-1700544973',
        db: 'admin'
      }
    ],
    authenticatedUserRoles: [ { role: 'readWrite', db: 'admin' }, { role: 'read', db: 'foo' } ]
  },
  ok: 1
}

Note that the authenticated user is now a dynamic user credential.

Templated Vault ACL policies provide great flexibility when defining access rules, but a single policy, or a specific group of policies, may not be enough to cover all use cases.

Creating additional Vault ACL roles for specific Nomad jobs can help better manage access control to Vault secrets.

Next steps

In this tutorial, you configured Nomad and Vault to communicate with ACL enabled. You also configured Nomad to automatically add workload identities for tasks that need access to Vault.

You then deployed Nomad jobs to read static and dynamic secrets from Vault using different Vault ACL roles and policies. Both jobs used their workload identities to receive a Vault ACL token properly scoped to the work they needed to do.

This process required several steps, with certain values having to match each other in order for everything to work properly.

There are two resources available to help you automate these steps.

You may continue exploring additional integrations between Nomad and Vault or learn how to similarly integrate Nomad and Consul with ACL enabled.

Collection Overview

Vault

 Next

Vault token generation